Discussion Forums
Discussion Forums > Category: Compute > Forum: AWS Lambda >Thread: Support for additional libraries?
Advanced search options
Support for additional libraries?
Posted by: rjocoleman
Posted on: Nov 14, 2014 11:48 AM
  Click to reply to this thread Reply
This question is not answered. Answer it to earn points.
Am I correct in my reading the only external library/dependency Lambda supports is ImageMagick ?
Are additional libraries planned or in the pipeline?

e.g. libav / avconv (https://libav.org/)

An example use case is transcoding media (bearing in mind the ephemeral disk limit) or extracting keyframes for use in thumbnails etc.
Permlink Replies: 15 | Pages: 1 - Last Post: Nov 20, 2016 11:23 PM by: PrasadG
Replies
Re: Support for additional libraries?
Posted by: Bradley@AWS
Posted on: Nov 16, 2014 12:26 PM
in response to: rjocoleman in response to: rjocoleman
  Click to reply to this thread Reply
ImageMagick is the only external library that is currently provided by default, but you can include any additional dependencies in the zip file you provide when you create a Lambda function. Note that if this is a native library or executable, you will need to ensure that it runs on Amazon Linux.
Re: Support for additional libraries?
Posted by: Eric Hammond
Posted on: Nov 18, 2014 4:49 PM
in response to: rjocoleman in response to: rjocoleman
  Click to reply to this thread Reply
rjocoleman:

I used my AWS Lambda shell hack to see the nodejs modules installed:

$ dirs=$(lambdash 'echo $NODE_PATH' | tr ':' '\n' | sort)
$ echo $dirs
/var/runtime /var/runtime/node_modules /var/task
 
$ lambdash 'for dir in '$dirs'; do echo $dir; ls -1 $dir; echo; done'
/var/runtime
node_modules
 
/var/runtime/node_modules
aws-sdk
awslambda
dynamodb-doc
imagemagick
 
/var/task # Uploaded in Lambda function ZIP file
lambdash.js
node_modules
Re: Support for additional libraries?
Posted by: Bryan.L@AWS
Posted on: Nov 21, 2014 10:07 AM
in response to: rjocoleman in response to: rjocoleman
  Click to reply to this thread Reply
Using other external libraries or binaries is possible with Lambda, however you will be limited by the max size of a Lambda function.

One way to achieve what you want, without using libav (as libav doesn't statically link very well), is to compile ffmpeg with static enabled and then shared disabled. You will need to compile and build this binary or any other dependencies on an Amazon Linux machine first.

./configure --enable-static --disable-shared

Once you've compiled ffmpeg, you can run a make, then copy the compiled ffmpeg binary into your Lambda function.

At this point you can use something like the following to generate thumbnails from your videos, or transcode.

process.env['PATH'] = process.env['PATH'] + ':' + process.env['LAMBDA_TASK_ROOT']
var ffmpeg = require('ffmpeg')
 
exports.ffmpeg = function(event, context) {
    new ffmpeg('./thumb.MP4', function (err, video) {
        if (!err) {
	        video.fnExtractFrameToJPG('/tmp', {
		    every_n_frames: 20,
	            file_name : 'image_frame_%t_%s'
	        }, function (error, files) {
	            if (!error)
	                console.log('Frames: ' + files);
 			context.done();	
	        });
        } else {
            console.log('Error: ' + err);
        }
    });
}


The important part of this is to make sure you add the task root path to your environment path, so that the ffmpeg NodeJS module can locate the binary. You could also install the fluent-ffmpeg NodeJS module and take advantage of the FFMPEG_PATH environment variable as well.

As for other Native Libraries, like OpenCV. You can try to statically compile all dependencies, and then when you run an NPM install of the Module in question, it will build the Native Module using the statically linked libraries. This removes any dynamic dependency requirements for the modules. However keep in mind that not all Modules will statically link, and not all dependencies will compile statically.

To recap, when compiling any binaries or dependencies, make sure you are doing so on an Amazon Linux machine and then copy the requirements to your Lambda Function.
Re: Support for additional libraries?
Posted by: taureon
Posted on: Nov 23, 2014 8:02 PM
in response to: Bryan.L@AWS in response to: Bryan.L@AWS
  Click to reply to this thread Reply
Tried the suggested approach. I'm seeing the following error:

Error: Command failed: /bin/sh: /var/task/ffmpeg: Permission denied

I'm using s3.getObject(params).createReadStream().pipe(file); to write the file to /tmp/infile.
Re: Support for additional libraries?
Posted by: taureon
Posted on: Nov 24, 2014 6:10 AM
in response to: taureon in response to: taureon
  Click to reply to this thread Reply
Worked out an alternative by moving ffmpeg to /tmp and changing permissions to on the binary at the start of the script. Seems to have done the trick.
Re: Support for additional libraries?
Posted by: Bryan.L@AWS
Posted on: Nov 24, 2014 7:58 AM
in response to: taureon in response to: taureon
  Click to reply to this thread Reply
This should work without the need to move ffmpeg to /tmp ( I have several working examples currently using ffmpeg NodeJS module as well as doing a child_process.spawn of the ffmpeg binary ). What were the permissions of ffmpeg in your ZIP before you upload it?
Re: Support for additional libraries?
Posted by: ambikaalex
Posted on: Jan 27, 2015 8:29 PM
in response to: Bryan.L@AWS in response to: Bryan.L@AWS
  Click to reply to this thread Reply
i am also facing same issue, here is the log

Logs
----
START RequestId: 3b650bc4-a6a5-11e4-93d9-09d3232868bd
2015-01-28T04:22:12.964Z 3b650bc4-a6a5-11e4-93d9-09d3232868bd Video file: Error: Command failed: /bin/sh: /var/task/ffmpeg: Permission denied

Process exited before completing request

END RequestId: 3b650bc4-a6a5-11e4-93d9-09d3232868bd
REPORT RequestId: 3b650bc4-a6a5-11e4-93d9-09d3232868bd Duration: 85.46 ms Billed Duration: 100 ms Memory Size: 1024 MB Max Memory Used: 11 MB

Message

Process exited before completing request::
Re: Support for additional libraries?
Posted by: Exvior
Posted on: Feb 5, 2015 12:06 AM
in response to: Bryan.L@AWS in response to: Bryan.L@AWS
  Click to reply to this thread Reply
I couldn't do anything to ffmpeg untill I moved it to /tmp and chmod it 755. I tried to chmod it in the working directory but was not going anywhere. I don't think Zip files have permissions saved with files.
Re: Support for additional libraries?
Posted by: Exvior
Posted on: Feb 9, 2015 12:47 AM
in response to: Bryan.L@AWS in response to: Bryan.L@AWS
  Click to reply to this thread Reply
I'm using ffmpeg to load video though the s3 url (http://s3.amazonaws.com/bucket/video.mp4).

I'm having an issue with ffmpeg trying to access the s3 video though the url too quickly. The lambda executes as soon as the file is put in the bucket, and it gets a 403 error until the 2nd attempt.

My solution was I had to create a setTimeout on that by 1 second that seems to solve that problem.

Bryan.L@AWS is this the right way to go about this?
Re: Support for additional libraries?
Posted by: The Bosco
Posted on: Feb 25, 2015 7:43 PM
in response to: Bryan.L@AWS in response to: Bryan.L@AWS
  Click to reply to this thread Reply
I'm currently trying to set up a Lambda function that uses ffmpeg to convert an animated .gif to an .mp4 that loops the .gif 4 times.

I'm in the middle of porting my bash/ffmpeg code to use the ffmpeg Node module instead of bash commands, but I'm starting to get nervous about the process of getting a compiled version of ffmpeg that will work with Lambda.

I'm currently just using S3 and spinning up an EC2 instance just to compile ffmpeg seems excessive, since several people have done it. Someone mentioned an NPM package that wraps a binary... Just wondering if that actually works or if I really need to compile ffmpeg on an Amazon server.
Re: Support for additional libraries?
Posted by: Sylvain Perron
Posted on: Feb 25, 2015 8:13 PM
in response to: The Bosco in response to: The Bosco
  Click to reply to this thread Reply
Hi The Bosco,
please have a look at my project Lambdaws, which has support of external libraries. Just make a pull request for ffmpeg (base yourself on the phantomjs) and it will be added in very small delay.

https://github.com/mentum/lambdaws#using-large-external-libraries

Your feedback would be appreciated
Re: Support for additional libraries?
Posted by: yogacentre
Posted on: Apr 9, 2015 2:41 PM
in response to: Bryan.L@AWS in response to: Bryan.L@AWS
  Click to reply to this thread Reply
I have to move ffmpeg into /tmp to get it working as well. I'd rather not do this, since I'm not sure how the lambda will act if the same container is reused and the file is already moved. Can you give any more info about how you got it working without moving it?
Re: Support for additional libraries?
Posted by: barrettharber
Posted on: Apr 13, 2015 11:10 AM
in response to: The Bosco in response to: The Bosco
  Click to reply to this thread Reply
You just need to use a binary that is pre-compiled for linux x86_64 architecture. See this for an example: https://github.com/binoculars/aws-lambda-ffmpeg
Re: Support for additional libraries?
Posted by: adamtravel
Posted on: Oct 18, 2015 2:38 AM
in response to: Exvior in response to: Exvior
  Click to reply to this thread Reply
@Exvior After 5 hours of debugging your post saved me from utter failure!

I tried a native 64-bit Linux binary of both PhantomJS and wkhtmltopdf -- each failed with a permission denied error when running from the base directory of my zip file.

Once I moved each executable to /tmp and did chmod 0775, they both worked perfectly.

Thank you!!

-1 Amazon for this ticket having been open for over 6 months!
Re: Support for additional libraries?
Posted by: PrasadG
Posted on: Nov 20, 2016 11:23 PM
in response to: adamtravel in response to: adamtravel
  Click to reply to this thread Reply
Hello,

I am trying to host a lambda function that takes in video url and generates thumbnail and store it to s3.
I am using FFmpeg . Is FFmpeg supported by lambda? or do i have to still go with the workaround moving static library files in /temp. if yes want to know if there are any issues caused due to this workaround please help?