Rails 4: gem install pg (when using the Postgresql App on OS X)

If you are getting a build error from Gem when trying to install the PostgreSQL gem called ‘pg’ and you are using the PostgreSQL App instead of installing PostgreSQL via Brew or some other method, it is because Gem cannot find the PostgreSQL config in the standard locations. To correct this you need to provide Gem with the path to the config file stored within the PostgreSQL App’s internal folder structure.

This can be done as follows:

gem install pg -- --with-pg-config=/Applications/Postgres.app/Contents/Versions/9.4/bin/pg_config

Another way to accomplish this is:

export CONFIGURE_ARGS=”with-pg-include=/Applications/Postgres.app/Contents/Versions/9.4/include/”
gem install pg

Note: the latest version of the PostgreSQL App, at the time of this writing was 9.4, your version may be different. Make sure you change the 9.4 in the config path in the command above to the one you are using or it will not work. You can verify the path by opening Finder, going to the Applications folder, locating Postgres.app, right clicking and selecting ‘show package contents’. This will open up the app’s folder structure in Finder.

Node.js: Simple Queuing for File Processing

QueueManager (JavaScript)

For the past few months I have been in the process of creating a custom, enterprise
grade, distributed media transcoding/encoding system for my employer. Because the
system is meant to be stand alone and can be integrated into any of the company’s
web products it needed to be extremely robust, meaning it needed to handle the
processing of any number of media files being passed to it at any given time and by
any number of clients. It is for this purpose that I used Node.js as the programming
platform. Node has the speed required for a system like this due primarily to its
non-blocking IO implementation.

Because the transcoder has to handle an unspecified number of videos added to it
at any given time, by any number of clients, it needed the ability to asynchronously
handle the addition of new media, queue that media for transcoding, while at the
same time be actively processing media that was previously queued. For this I needed
a simple and easy to understand queuing mechanism. Below is what I came up with.

The code below was written to deal with the processing of a file at the time it
is added to a directory being watched by the transcoding service. To put it simply:

  1. A Node file system watcher, such as Chokidar, watches a specified directory
  2. Each time a file is added to that directory it is wrapped in a ‘File Processor’
    (described below) that is responsible for processing that file type
  3. The processor is then added to the ‘QueueManager’ (described below). The manager places the processor in the primary queue
  4. If the manager is currently processing, files nothing happens after the processor is added to the queue
  5. If the manager is not processing, it will iterate over the main queue removing processors in the main queue and placing a reference to each processor’s ‘process’ method into a processing queue (see Async File Processor below)
  6. Once the main queue has been emptied, the processing queue is passed to the Async library for processing in either a series (one at a time) or parallel (each processor spun off onto its own sudo-subprocess) – see Async Library for details on implementation

Possible Issues:

Question: What happens if the main queue is constantly being added to while the manager is trying to empty it?

Answer: if there is no halt in main queue additions an infinite loop would occur.

Likelihood: Extremely low considering the system and its use.

/**
 - Created by Tim on 5/28/15.
 */

var async = require('async'),
    logger = require('./logger'),
    _ = require('underscore');

var component = 'QueueManager: ';
var queue = [];
var isProcessingQueue = false;
var isParallelProcessingActive = false;

// Event handler called when Async is done processing the current queue
function queueProcessingComplete(err, results){
    isProcessingQueue = false;

    if(err){
        logger.error(component + 'Error Processing queue: ERR-MSG: ' + err.message);
    }else{
        processQueue();
    }

}

// Processes the queue by moving all current items in the queue
// into a temporary list which is then handed to the Async library
// for processing in either series or parallel depending on need
function processQueue(){
    if(!isProcessingQueue){
        var managerList = [];
        var shifted = undefined;
        var flag = true;

        while(flag) {
            shifted = queue.shift();
            if(_.isUndefined(shifted)){
                flag = false
            } else {
                managerList.push(shifted.process);
            }
        }

        if(managerList.length > 0) {
            isProcessingQueue = true;
            // Execute each of the Transcoders in series
            if(isParallelProcessingActive){
                async.parallel(managerList, queueProcessingComplete);
            } else {
                async.series(managerList, queueProcessingComplete);
            }
        }
    }
}

// Public function for adding 'processors' to the queue
exports.pushToQueue = function(processor){
    queue.push(processor);
    processQueue();
};

Async File Processor (JavaScript)

The Async library requires a processing function to be passed to it that accepts a callback
function. The callback is used by the processing function to notify Async of
either its success or failure in performing its job. For that reason I have created
the ‘Processor’ template below.

By using a JavaScript ‘Constructor Function’ to create a new object that wraps the
Async processing function, one can maintain a reference through ‘that’ to any and
all relevant objects and data necessary for not only the processing of the file but
for returning results via the Async callback function.

/**
 * Created by Tim on 5/27/15.
 */

var fs = require('fs-extra'),
    logger = require('./../utils/logger'),
    _ = require('underscore');

var component = 'Processor: ';

// Constructor Function: creates a new unique processor
function Processor(file){
    // Allow reference to instance in callbacks
    var that = this;

    //'Private' Variables
    this._file = file;
    this._active = true;
    this._completeCallback = undefined;

    //'Private' Methods

    //'Event Handlers' Methods
    this._handleProcessingComplete = function (err, results) {
        if(err){
            var msg = component + 'File processing did not complete successfully' +
                '. ERR-MSG: ' + err.message;

            logger.error(msg, null);

            // Your processing error code here

            // End processing and hand execution back to queue
            that._completeCallback(null, new Error(msg));

        } else {

            // Your processing complete handling code here

            // End processing and hand execution back to queue
            that._completeCallback(null, results);
        }
    };

    //'Public' Methods
    this.process = function (callback) {
        // Allow notification of processing completion
        that._completeCallback = callback;

        // Prepare for file processing code here

        // Only execute processing task if initialization was successful
        if(that._active){

            // Non-blocking IO File processing code here that eventually calls the
            // that._handleProcessingComplete function which in turn calls the
            // Async callback function to notify processing is complete

        } else {
            // End processing and hand execution back to queue
            that._completeCallback(null, []);
        }
    };
}

module.exports = Processor;

Conclusion

That about does it, a simple queuing mechanism for Node.js file processing. As always
any feedback or suggestions are welcome so long as they are constructive.

Sublime Text 3: Hide Hidden Files and Folders in Sidebar

When using Sublime Text 3 on my Mac I noticed that it was showing all hidden files and folders in the left-hand folder/file navigation pane. This can become quite annoying in a large project with tens of files and folders, specially when it shows all of the ._ files for each and every file in the project. To stop this from happening all you need to do is edit your .sublime-project file so it looks similar to the following.

{
    "folders":
    [
        {
            "path": ".",
            "folder_exclude_patterns": [".*"],
            "file_exclude_patterns": [".*"]
        }
    ]
}

As soon as you save the project file the hidden files and folders will instantly vanish from the left-hand pane.

Node.js + Chokidar: Wait for file copy to complete before modifying

If you have ever spent any time dealing with folder watchers in pretty much any language, you will probably have noticed that the watcher usually notifies your application of a new file added to the folder the instant the file is added to the folder’s index. This however does not mean that the file is complete, it may still be in the process of being copied or saved to the disk. This creates a problem when you are watching for new files so that they can then be processed in some manner. To get around this you will need to check the API for the language/platform you are using to find out if the IO library has a way to check if a file is whole before processing.

I ran into this same issue while building a bulk video file transcoding system on top of the Node.js JavaScript platform. Unfortunately I could not find a built in method for checking if a file was completely saved before acting on it and so had to handle this myself. The following assumes you know how to use Chokidar or some other Node file watcher package from NPM. The function of interest is titled ‘checkFileCopyComplete’, hopefully this will help speed things up if you are looking for this solution.

// Setup video source folder observer for notifications of new files
var chokidar = require('chokidar');

var watcher = chokidar.watch(config.videoRawFolder, {
    persistent: true,
    followSymlinks: false,
    usePolling: true,
    depth: undefined,
    interval: 100,
    ignorePermissionErrors: false
});

watcher
    .on('ready', function() { logger.info('Initial scan complete. Ready for changes.'); })
    .on('unlink', function(path) { logger.info('File: ' + path + ', has been REMOVED'); })
    .on('error', function(err) {
        logger.error(component + 'Chokidar file watcher failed. ERR: ' + err.message);
    })
    .on('add', function(path) {
        logger.info('File', path, 'has been ADDED');

        fs.stat(path, function (err, stat) {

            if (err){
                logger.error(component + 'Error watching file for copy completion. ERR: ' + err.message);
                logger.error(component + 'Error file not processed. PATH: ' + path);
            } else {
                logger.info(component + 'File copy started...');
                setTimeout(checkFileCopyComplete, fileCopyDelaySeconds*1000, path, stat);
            }
        });
    });

// Makes sure that the file added to the directory, but may not have been completely copied yet by the
// Operating System, finishes being copied before it attempts to do anything with the file.
function checkFileCopyComplete(path, prev) {
    fs.stat(path, function (err, stat) {

        if (err) {
            throw err;
        }
        if (stat.mtime.getTime() === prev.mtime.getTime()) {
            logger.info(component + 'File copy complete => beginning processing');
            //------------------------------------- 
            // CALL A FUNCTION TO PROCESS FILE HERE
            //-------------------------------------
        }
        else {
            setTimeout(checkFileCopyComplete, fileCopyDelaySeconds*1000, path, stat);
        }
    });
}

PHP/Laravel: Input::all() and Input::file(‘files’) coming back as empty arrays when uploading large files (videos/images)

Are you trying to implement an image or video uploader in your PHP project, specifically a Laravel 4+ project, and you find that the upload HTTP POST is indeed hitting your controller method but Input::all(), Input::file(‘files’), and $_FILES are all empty arrays? Well there is a really good chance its not your front-end nor your back-end code that is the problem, its most likely its PHP playing a cruel trick on you. According to the PHP documentation, if the php.ini default settings are in play, PHP is restricting file uploads that are bigger than the default size of 8 megabytes.

Yes that is correct, in today’s modern world of web development and today’s internet of media, the makers of PHP have decided that file uploads should still be restricted to a file size more appropriate for the late 1990’s. But on top of that, instead of providing a warning message in the HTTP Request object that the limit is being exceeded, PHP just simply removes the request objects all together and pretends nothing was submitted with the request at all, leaving the developer in a complete state of profound confusion.

That being said, go change your defaults (CentOS specific):

nano /etc/php.ini

Search for each of these default keys and set the values you feel best fit your needs:

memory_limit = 4000M
upload_max_filesize = 1000M
post_max_size = 1500M

Restart the Apache service:

service httpd restart

Building JWPlayer v6.12 from Source (for the rest of us)

Building JWPlayer v6.12 from Source with an Embedded RTMPE Secret Key for use with Protected Wowza Server (for the rest of us)

Last Updated: 04/29/2015

Used to rebuild the current version of the player with an embedded secret code for use with the Wowza RTMPE stream security protocol. Must be built on a machine that can run old 32-bit versions of Java specifically the 1.5 JDK (this rules out OS X after Mavericks as Apple does not support it).

Build Machine Setup

Requirements

  • 32-bit Windows XP/7/8.1
  • 32-bit Java 1.5
  • Ant 1.7.0
  • Adobe Flex SDK 4.5.1

Installing Java 1.5

URL: http://www.oracle.com/technetwork/java/javasebusiness/downloads/java-archive-downloads-javase5-419410.html

  1. Set the JAVA_HOME environment variable to the installation location of Java JDK root folder (usually C:\Program Files\Java\jdk1.5.0_22)
  2. Add the bin directory of the Java JDK to your system’s PATH environment variable by referencing the JAVA_HOME environment variable, this makes it so there is only one place that the Java JDK path needs to updated (%JAVA_HOME%\bin)
  3. Close all Command Prompt windows if any are open as they will not have the changes to the path you made until reopened
  4. Verify the system is now using Java 1.5 by opening a clean Command Prompt and executing the command java -version

Installing Apache Ant 1.7.0 (open build platform used in the Java world)

URL: http://archive.apache.org/dist/ant/binaries/

  1. Download the binary zip and extract it
  2. Copy the extracted folder to the C drive (C:\apache-ant-1.7.0)
  3. Add the Ant bin folder to the Windows PATH environment variable (C:\apache-ant-1.7.0\bin)
  4. Close all Command Prompt windows if any are open as they will not have the changes to the path you made until reopened
  5. Verify the system is now using Ant 1.7 by opening a clean Command Prompt and executing the command:

ant -version

Installing Flex SDK 4.5.1

URL: http://sourceforge.net/adobe/flexsdk/wiki/Download%20Flex%204.5/

  1. Create a new folder in the C drive called JWPlayer (C:\JWPlayer)
  2. Download the SDK zip archive and extract the contents to the JWPlayer folder (C:\JWPlayer\flex_sdk_4.5.1.21328A)

Installing the JWPlayer Source

URL: https://github.com/jwplayer/jwplayer/tree/release

  1. Download the 6.12 release of the JWPlayer source code (at this time it is the current release and is therefore in the release branch on GitHub, this will change as new releases come out)
  2. Extract the archive to the JWPlayer folder you created early (C:\JWPlayer\jwplayer-release-6.12)

Building the Player from Source

Update Build Properties

  1. Open the build.properties file located in the JWPlayer source folder you just extracted (C:\JWPlayer\jwplayer-release-6.12\build\build.properties)
  2. On line 11 change the ‘flexsdk’ key with the location of the Flex SDK, you must use Unix style file path (flexsdk = C:/JWPlayer/flex_sdk_4.5.1.21328A)
  3. On line 16 change the ‘execextension’ key with a ‘.exe’ as we are using Windows to build (execextension = .exe)
  4. Save and close the file

Update Player Version

  1. Open the PlayerVersion.as file (C:\JWPlayer\jwplayer-release-6.12\src\flash\com\longtailvideo\jwplayer\player\PlayerVersion.as)
  2. On line 5 change the value “JWPLAYER::version” to “6.12.0”

Adding the Secret Key

  1. Open the RTMPMediaProvider.as file (C:\JWPlayer\jwplayer-release-6.12\src\flash\com\longtailvideo\jwplayer\media\RTMPMediaProvider.as)
  2. On line 491 change var hash:String = TEA.decrypt(evt.info.secureToken, getConfigProperty(‘securetoken’)); to var hash:String = TEA.decrypt(evt.info.secureToken, “VALUE_OF_NEW_SECRET_KEY_GOES_HERE”);

Building

  1. Open a Command Prompt at the root of the JWPlayer’s source folder (C:\JWPlayer\jwplayer-release-6.12)
  2. Execute the command

ant -buildfile build\build.xml

Troubleshooting

So many different things can go wrong, way too many to list in this document, Google is a great resource 😉

Release Binaries

New player should be located in bin-release if the build was successful
Assets Created jwplayer.js, jwplayer.html5.js, jwplayer.flash.swf

Notes:

Adding a JAVA_HOME environment variable to your system and adding Java to your PATH by referencing JAVA_HOME, instructions can be found here https://kaanmutlu.wordpress.com/2013/04/26/setting-the-java_home-in-windows-7-64-bit/

2.23 – Apps must follow the iOS Data Storage Guidelines or they will be rejected [FIXED]

If you have run into an iOS application rejection based on the 2.23 iOS guideline regarding an application exceeding fair use of the iCloud backup storage then you will need to make sure all files your application downloads or creates, that are not user specific and that do not need to be backed up, are removed from the iCloud backup set for the application. This can be done by applying the NSURLIsExcludedFromBackupKey key to all of those files using the following method.

You can add this method to your AppDelegate or some other singleton within your application and have access to it from anywhere within your project using the following code:

// Make sure file is removed from the iCloud image backup system
AppDelegate *appDelegate = [[UIApplication sharedApplication] delegate];
[appDelegate addSkipBackupAttributeToItemAtURL:imagePath];

// Removes all application assets that are not user created from the iCloud backup system
// by adding the NSURLIsExcludedFromBackupKey to every file that is downloaded and stored
// in the application's document directory
-(BOOL)addSkipBackupAttributeToItemAtURL:(NSString *)fileString {
    
    // Convert string path to NSURL
    NSURL * fileURL = [NSURL fileURLWithPath:fileString];
    
    // Make sure file has already been created before applying key
    if (![[NSFileManager defaultManager] fileExistsAtPath:[fileURL path]]) {
        NSLog(@"File Missing: %@",[fileURL path]);
        return NO;
    }
    
    // Obtain iOS version for proper key application
    NSString *currSysVer = [[UIDevice currentDevice] systemVersion];
    
    // 5.0.1 requires <sys/xattr.h>
    if ([currSysVer isEqualToString:@"5.0.1"]) {
        const char* filePath = [[fileURL path] fileSystemRepresentation];
        const char* attrName = "com.apple.MobileBackup";
        u_int8_t attrValue = 1;
        int result = setxattr(filePath, attrName, &attrValue, sizeof(attrValue), 0, 0);
        NSLog(@"Excluded '%@' from backup",fileURL);
        return result == 0;
    }
    // Above 5.0.1 use NSURLIsExcludedFromBackupKey key
    else if (&NSURLIsExcludedFromBackupKey) {
        NSError *error = nil;
        BOOL result = [fileURL setResourceValue:[NSNumber numberWithBool:YES] forKey:NSURLIsExcludedFromBackupKey error:&error];
        if (result == NO) {
            NSLog(@"File '%@' could not be excluded from iCloud backup set. Error: %@",fileURL, error);
            return NO;
        }
        else {
            NSLog(@"File '%@' has been removed from iCloud backup set.",fileURL);
            return YES;
        }
    } else {
        // iOS version is below 5.0, no need to do anything
        return YES;
    }
}