Google recently released their own (beta) version of AWS Lambda functions: Google Cloud Functions.

I realize that AWS has a service that does transcoding (link), but I already had a gcloud project set up, and thought it would be fun to see what GCF could do for me.

CLI dependencies - gcloud - nodejs

Fun with functions!

Set up the function

Functions are pretty simple. They have a package.json for tracking dependencies, and a function file (index.js or function.js).

Create a folder for the files to live in, and make a file named index.js. Run npm init in this folder to create a package.json.

Skipping ahead, here's how my directory structure ended up.

directory structure

In index.js, just send some text so you can confirm that it's working.

index.js

  module.exports.transcode = (req, res) => {
    res.send('howdy!')
}

Deploy the function

Before we get too far, I like to make sure everything is set up for the function to actually work. Because GCF is still in beta, you will need to enable the API manually. Go to the cloud console and click some buttons!

You will need a cloud storage bucket to stage the function code. Make a bucket here, or use an existing one.

Now you can deploy! For example:

gcloud beta functions deploy transcode --local-path=./ --trigger-http --stage-bucket=buckey-mc-bucketface

I'm using --trigger-http so the function will be exposed as a REST endpoint. I'm going to be calling it over http, but you can also call directly from the command line.

Run gcloud beta functions call transcode to test it out.

Docs

Run locally

Deployment takes a few minutes every time, which really sucks as a development workflow. Time to set up a local dev environment! Here's a simple express server that accepts POST requests to /transcode, just like the endpoint that we got from gcloud.

dev-server.js

  #!/usr/bin/env node

const express = require('express')
const app = express()
const router = express.Router()
const transcodeFunction = require('./transcode/index')

router.post('/transcode', transcodeFunction.transcode)
app.use('/', router)

// start the server
const port = 3030
const http = require('http')
app.set('port', port)
const server = http.createServer(app)
server.listen(port)

Now we can write a test case to post to the endpoint. It's going to accept a video file in the formdata.

test.js

  #!/usr/bin/env node

const fs = require('fs')
const request = require('request')
const argv = require('yargs').argv
const url = argv.dev ? 'http://localhost:3030/transcode' : 'https://us-central1-your-project-name.cloudfunctions.net/transcode'

request.post(url, {formData: {

    video: fs.createReadStream(`${__dirname}/../test.webm`)

}}, (err, res, body) => {
    if(err) {
        return console.error(err, res, body)
    }
    if(typeof body === 'string') body = JSON.parse(body)
    fs.writeFile('../tmp/buffer.mp4', Buffer.from(body.buffer.data))    
})


Once you have the dev server started, you can run node ./test.js --dev to run the test case locally. Nothing is happening with the video file yet, but we'll need it once we start writing the function. Similarly, we need to save the result to a file (maybe comment out for now).

Transcoding

Installing FFmpeg

The difficulty here is that the FFmpeg binary is compiled differently depending on what system it's running on. Fortunately, there's a way to automatically install the correct FFmpeg binary, depending on the system.

Introducing ffmpeg-installer, a platform independent binary installer of FFmpeg for node projects. Add it to your package.json! npm install --save ffmpeg-installer

Use the file system? No.

Hey, neat! I can run FFmpeg, but I can't write anything to disk. This is one of the limitations of this serverless environment. Totally different from how I usually work, but thankfully there is a way to use streams and buffers to avoid writing to disk.

I'm honestly not sure how to pass streams to spawned processes, but fluent-ffmpeg accepts streams for inputs and outputs, so I used it.

Here's my final function file.

index.js

  
const ffmpegPath = require('@ffmpeg-installer/ffmpeg').path
const FfmpegCommand = require('fluent-ffmpeg')
FfmpegCommand.setFfmpegPath(ffmpegPath)
const fs = require('fs')
const path = require('path')
const stream = require('stream')
const multer = require('multer')
const storage = multer.memoryStorage()
const mult = multer({ storage })

class WritableChunkCache extends stream.Writable {

    constructor(options) {
        super(options)
        this._chunks = []
    }

    write(chunk, encoding, callback) {
        this._chunks.push(chunk)
        if(typeof callback === 'function') callback()
    }

    writev(chunks, callback) {
        this._chunks = this._chunks.concat(chunks)
        if(typeof callback === 'function') callback()
    }

    get buffer(){
        return Buffer.concat(this._chunks)
    }
}

exports.transcode = function(req, res){

    // handle multipart form data
    var videoUpload = mult.single('video')
    videoUpload(req, res, () => {

        var file = req.file
        var readStream = new stream.PassThrough()
        readStream.end(new Buffer(file.buffer))
        var outStream = new WritableChunkCache()

        var command = new FfmpegCommand(readStream)
            .fps(30)
            .on('end', () => {
                res.send({buffer: outStream.buffer})
            })
            .on('error', (err, ...args) => {
                console.error(err, args)
                res.send(err)
            })
            .format('mp4')
            .videoCodec('libx264')
            .outputOptions([
                '-movflags frag_keyframe+empty_moov' // accounting for unknown duration due to streamed input
            ])
            .pipe(outStream, { end: true })
    })
}

In order to send the data in the response, I needed to get the buffer from the output stream. By default, the buffer is consumed by the stream, not cached. I wrote a dead simple stream implementation (WritableChunkCache) that caches the chunks, and has a gettable buffer.

It almost works!

For testing, I was using a small webm file; only 1.5MB. That worked swimmingly! I transcoded a webm into an mp4, and saved it into a bucket. Nice!

In the real world, you probably don't need to transcode tiny video files. GCF has some severe limitations that I should have read about before getting this far. There is a 10MB upload limit. It won't tell you this if you try uploading a file that is >10MB. It just fails and sends you an empty string. Thanks, Google.

Beware the quotas

Failure!

In conclusion... don't do this. It doesn't work. Delet this. I hope you learned something.

All of the FFmpeg stuff above can be used on appengine. Do that, or use AWS elastic transcoder.


10,280 7 1