Create a symlink to the last jpeg image , also fixed a raced condition

Description of Patch

        "last_jpeg_filename",
        "# File name for the symlink to the last saved image. Example: lastimage\n"
        "# The (.jpg|.ppm) extension will be added to the filename you specified.\n"
        "# Default: undefined\n",

It is very similar to the lastsnap.jpg feature.

The user could put "last_jpeg_filename lastimage" in the conf and motion will create targetdir/lastimage.jpg as a symlink to the last saved image. To disable this feature, the user should just comment out the line in the config.

The code is in event.c , in event_image_detect, just after the put_picture call.

While I was there, I also corrected a race condition:

unlink(symlink);
// in this interval the file will be missing, causing errors to apps always expecting an image here
symlink(target,symlink);

A better way to do this is using 'rename', it overwrites the target by default and is truely atomic:

symlink(target, temporary_name);
rename(temporary_name, symlink); 

Installation of Patch

Download the patch file. If it is packed as a gz or tar.gz unpack it first. Then copy it to the motion source directory and issue the command (assuming the patch file is called filename_of_patch_file.diff)

patch < filename_of_patch_file.diff

Then re-build Motion and test the patch.

Change History of Patch

• 1.0 Initial revision

Discussion and Comments

---

The current symlink feature related to the periodic webcam feature (snapshot) only creates a link maybe once per minute or less.

This feature will cause maybe 30 links to be created per second.

However if we created the link once an event it over then the load becomes much smaller. But then you have to wait maybe minutes for a website to be up to date.

I never liked the current snapshot feature too much with its hardcoded name and not being able to turn it off. So I am very open to a better feature. But e.g. I run 14 camera threads. If I have them running at high frame rate and more than one sees motion then I double the disk activity which is already very high. Such a feature will not work well.

So we have to come up with a better solution.

The use case is - I suppose - to have a good image on a website representing the latest action seen by the camera.

Maybe we can brainstorm a little on it.

• We can symlink first image in an event
• We can symlink the last image in an event
• We can symlink the best image in an event (most motion - we already have a similar feature)
• We can symlink every 10 seconds in long events.

The idea being that the symlink is not so often made but it always points to something giving the best impression of the current or last event in one picture. But without adding a huge file i/o load.

Any other ideas? Any comments?

-- KennethLavrsen - 13 May 2007

---

I already use this feature in production . Our setup is a bit complicated:

- We have live streaming created by a http daemon and played with much javascript hackery. When the viewer is in a LAN we have 2-3FPS . This also supports custom composed views (in a 2x2 .. 5x5 grid) - We have a distributed system . 20 Satellites in different locations constantly upload all jpegs to a central storage machine. For years we tried different variants like keeping an sql queue and using forked event handlers, but this symlink feature is by far the best. We constantly walk the filesystem tree between the last uploaded and the last saved and upload to the central machine using a custom (very slim) protocol. The central machine has the streaming feature above, and most of the time the user can sit at home , log in to the company surveillence app, and view composed grids with all his setups -almost realtime-.

You can see how constant polling (evil people call it busywait ) revealed the race condition. Happily rename has come to the rescue:

If newpath already exists it will be atomically replaced (subject to a few conditions; see ERRORS below), so that there is no point at which another process attempting to access newpath will find it missing.

Now, if we could be able to use some "tweak" parameter so we can have both low IO AND support setups like I described, it would be great. But this is better discussed on IRC, cause brainstorming is always better there.

Thanks a lot for a great project. Great code quality as well, I was able to imediatly dive into the code , that's always cool.

-- BogdanLucaciu - 14 May 2007

---

I agree that your change that uses rename to avoid the race condition should be implemented no matter what.

And running a system at 2-3 fps is not an issue either. I run a small perl script that rebuilds by website each time a jpeg is saved. This is a forked process and it still works reasonably well. But some users run Motion at 30 fps and they have major performance issues to keep up this framerate with more than 1-2 cameras. And it is for them I want to implement the symlinking in a more efficient way than just creating and renaming 30 times per second for each camera.

I assume your application does not require a symlink for each jpeg saved.

-- KennethLavrsen - 14 May 2007

---

I made a small benchmark just to measure the I/O overhead. The running code is very similar to the equivalent C code, so I'm confident the benchmark is accurate

Benchmark: timing 10000 iterations of No symlink, With symlink...
No symlink: 22 wallclock secs ( 1.59 usr +  2.68 sys =  4.27 CPU) @ 2341.92/s (n=10000)
With symlink: 32 wallclock secs ( 1.72 usr +  2.90 sys =  4.62 CPU) @ 2164.50/s (n=10000)

So basically we have 454jpeg/s vs 312jpeg/s

Machine info

• model name : Intel(R) Celeron(R) CPU 2.80GHz
• Memtotal: 1034936 kB
• It's a server that also does other stuff

Code:

#!/usr/bin/perl
use strict;
use warnings;
use Benchmark;
use File::Path;
my $image = 'image' x (10000); # 50k scalar :)
my $k = time; #counter
timethese (10000, {
        "No symlink"=> sub {
                saveimage();
        },

        "With symlink" => sub {
                my $imgpath = saveimage();
                symlink($imgpath,'lastimg.jpg.tmp');
                rename ('lastimg.jpg.tmp','lastimg.jpg');
        },

});

sub saveimage {
        $k++;
        my ($dir, $file) = imgpath($k);
        mkpath($dir);
        open my $fh, '>', $file;
        print $fh $image;
        close $fh;
        return $file;
}

sub imgpath {
        my $time = shift;
        my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime $time;
        my $dir = sprintf '%04d/%02d/%02d/%02d/%2d', 1900+$year, 1+$mon, $mday, $hour, $min;
        return ($dir, sprintf "$dir/%02d.jpg", $sec);
}

-- BogdanLucaciu - 19 May 2007

---

I am not sure a perl program is a good benchmark. But it actually shows a significant slowdown.

But please consider the case of having 5 cameras running at 30 fps in a small store. The original feature will now have to create in addition to the jpegs 150 symlinks per second + 150 renames per second + 150 file deletions.

As it is today even a fast machine looses frames if you use too large pre_capture values. I am very focussed on keeping the performance as high as possible.

I cannot support a feature that lowers performance for the most typical users of Motion. Especially when you consider that out of these 150 links per second you end up using one out of maybe 10000 of them for a still picture image.

I support adding a better linking to a good image for a website but it has to be done in an intelligent way and I suggested possible solutions above which creates safe links but not more often than Motion could manage even with many parallel cameras at high framerates.

-- KennethLavrsen - 04 Jul 2007

---

• last_jpeg_filename.patch: patch against r184 trunk