Cacomania

Cacomania

Develop a Firefox add-on using GhostText as example

Guido Krömer - 652 days ago - Tags: , ,

During the last months it might seem that my blog is dead, but it is not. I spend my most open source time developing GhostText. I already mentioned it here SublimeTextArea, which was the old name this project. But Federico and I decided renaming the project since it does not have to be restricted to Sublime Text. If you have no idea what GhostText is I recommend this video which show the the Chrome extension version.

read more

Comment

Kill all long running MongoDB queries at once

Guido Krömer - 732 days ago - Tags: ,

If of some very running queries killing your MongoDB server you should try to execute the following piece of code in the shell. It kills all long running queries at once. You can define the maximum execution time by using the first parameter, which is set to 120 seconds by default.

(function (sec) {db.currentOp()['inprog'].forEach(function (query) { 
    if (query.op !== 'query') { return; } 
    if (query.secs_running < sec) { return; }  

    print(['Killing query:', query.opid, 
           'which was running:', query.secs_running, 'sec.'].join(' '));
    db.killOp(query.opid);
})})(120 /*The maximum execution time!*/);

This is the output generated by my little emergency helper:

Killing query: 598841625  which was running: 1892 sec.
Killing query: 598818776 which was running: 1931 sec.
Killing query: 598771338 which was running: 2048 sec.
Killing query: 577527534 which was running: 73569 sec.
Killing query: 577589300 which was running: 73404 sec.
Killing query: 577540188 which was running: 73535 sec.
Killing query: 598254342 which was running: 3114 sec.
Killing query: 597396058 which was running: 4811 sec.
Killing query: 597609629 which was running: 4382 sec.
Killing query: 597403975 which was running: 4801 sec.
Killing query: 597423364 which was running: 4766 sec.
Killing query: 597389768 which was running: 4842 sec.

I hope my little posting helped you, please let me know if you liked or disliked it.

Comment

MongoDB with PHP "… BadValue $in needs an array"

Guido Krömer - 791 days ago - Tags: ,

Recently I had a problem when using MongoDB with PHP using the $in operator which ends up with the following error: "Can't canonicalize query: BadValue $in needs an array".

I did a simple query and used the $in operator thousand times before, but what went wrong? Let's take a look at the simplified code below, an indexed array of tags is given to filter for blog posts in a collection, but before passing the tags to the query I removed invalid tags from the array using unset().

foreach ($tags as $key => $tag) {
    if (!valid($tag)) {
        unset($tags[$key]);
    }
}

$posts = $blogPostCollection->find(['Tag' => ['$in' => $tags]]);

Using unset was the problem, but why? When iterating over an array after unsetting a value everything seems to be right:

php > $ids = ['a', 'b', 'c', 'd'];
php > unset($ids[1]);
php > foreach ($ids as $id) { echo "$id "; }
a c d 

The problem is that unset() produces holes in an indexed array, c has still the index two… :

php > $ids = ['a', 'b', 'c', 'd'];
php > var_export($ids);
array (
  0 => 'a',
  1 => 'b',
  2 => 'c',
  3 => 'd',
)
php > unset($ids[1]);
php > var_export($ids);
array (
  0 => 'a',
  2 => 'c',
  3 => 'd',
)

One solution is using array_splice() instead of unset() which prevents holes in an index array:

php > $ids = ['a', 'b', 'c', 'd'];
php > array_splice($ids, 1, 1);
php > var_export($ids);
array (
  0 => 'a',
  1 => 'c',
  2 => 'd',
)

Or you could use array_values() on the array after an unset operation which creates an new array containing all values from the origin array.

php >$ids = ['a', 'b', 'c', 'd'];
php > unset($ids[1]);
php > var_export(array_values($ids));
array (
  0 => 'a',
  1 => 'c',
  2 => 'd',
)

I hope this little post helped you with this or a similar problem, feel free to leave a comment.

Comment

SublimeTextArea

Guido Krömer - 822 days ago - Tags: , ,

Recently I wrote about "Get rid of those textarea with Chrome and Sublime Text 3" a Chrome extension which connects a HTML textarea through a WebSocket with a Sublime Text 3 plugin, for live editing the textarea's content in a decent editor.

After a request on GitHub, I finally released my SublimeText 3 plugin at Package Control and created a packed Chrome extension for easily installing at Chrome/Chromium.

So check it out if the browser textarea bothers you, too. Fell free to leave a comment here if you like or dislike my little extension.

Comment [1]

Get rid of those textarea with Chrome and Sublime Text 3

Guido Krömer - 898 days ago - Tags: , , , ,

My recent project is a proof of concept connecting a HTML textarea with Sublime Text 3 using WebSockets, which lets you use all nice features of Sublime Text in a common textarea. Therefore I wrote a Chrome extension and a Sublime Text 3 plugin, the Sublime Text plugin starts a simple, single threaded, WebSocket Server awaiting a connection from the Chrome extension for sending and receiving the text when it has been changed.

I made a short video of this concept, which can be watched at YouTube.

read more

Comment [6]

Test your RESTful API with Frisby

Guido Krömer - 910 days ago - Tags: ,

Writing a RESTful API is easy, but ensuring that small changes/fixes to the API does not break compatibility with existing clients consuming the API is not easy.

Not easy without functional tests, with the usage of a framework specified in REST API testing like Frisby even the tests are written easily. Frisby is build on top of Jasmine a node.js based BDD framework.

read more

Comment [5]

Unix timestamp conversion with date

Guido Krömer - 990 days ago - Tags: ,

If you are not smart enough to interpret a Unix timestamp by yourself date can do it for you.

$ date -u -d @1234567890
Fri Feb 13 23:31:30 UTC 2009

The timestamp can be formatted, too.

$ date -u -d @1234567890 +"%Y-%m-%d %T"
2009-02-13 23:31:30

If you need a timestamp, %s lets date print one.

$ date +"%s"
1386708117

Comment

Simple AngularJS sum by key view filter

Guido Krömer - 999 days ago - Tags: ,

This little view filter for AngularJS sums up the values of a spcific field from an array of objects.

This is quite useful if you want to show the total number of something at a table footer, for example.

angular.module('caco.feed.filter', [])
    .filter('sumByKey', function() {
        return function(data, key) {
            if (typeof(data) === 'undefined' || typeof(key) === 'undefined') {
                return 0;
            }
 
            var sum = 0;
            for (var i = data.length - 1; i >= 0; i--) {
                sum += parseInt(data[i][key]);
            }
 
            return sum;
        };
    });

Here is a small fiddle which shows how the filter works:

Let me know if you liked or disliked my little posting.

Comment [12]

Remove duplicate values from a mysql table

Guido Krömer - 1036 days ago - Tags:

I needed a way to clean up a MySQL table containing many duplicate rows because of a missing UNIQUE KEY.

One solution would be a script which searches and removes those dups, but depending on the number of rows this would be a heavy lifting job. A better solution is performing this task completely in the database without moving the data into an application and the sanitized data back to the database.

You do not need any PL/SQL…, five simple SQL statements can handle the task. A new table with the structure of the table containing the dubs, with a unique index preventing the dups, has to be created. The rows from the old table has to be inserted into the new one, by using the IGNORE keyword the query will not stop when reaching a duplicate entry, those dups gets just ignored. The last step is replacing the new table with the old one.

CREATE TABLE `table_without_dups` LIKE `my_table`;

ALTER TABLE `table_without_dups`
ADD UNIQUE `my_unique_key` (`col_1`, `col_n`);

INSERT IGNORE INTO `table_without_dups`
SELECT * FROM `my_table`;

DROP TABLE `my_table`;
RENAME TABLE `table_without_dups` TO `my_table`;

I hope my small posting helped you, feel free to leave a comment.

Comment

A simple AngularJS client side pagination module

Guido Krömer - 1182 days ago - Tags: ,

After my last tutorial, I wanted to do some more complex with AngularJS and solve my news feed problem in a similar way I solved my bookmark chaos. During this, I ran into the problem that displaying large lists, containing hundred of entries, can slow down the app significant using ng-repeat. On slow browsers like the Firefox or Opera the application hangs for some seconds during rendering the list of feed items. Since the JSON encoded and GZIP compressed list with 1.000 feed items for example is transferred quite fast, I did not want to do the pagination on the server side instead of this I wanted to paginate the list on the client. Doing the pagination on the client has the benefit that other filters could be used at the same time, too.

Client side pagination with AngularJS

read more

Comment [35]

A simple RESTful AngularJS bookmark app

Guido Krömer - 1208 days ago - Tags: , ,

Welcome to the second part of my AngularJS tutorial, you can read the first part here which covers a RESTful API build with the Slim PHP micro framework. AngularJS is a JavaScript framework for building single page applications in a MVC style. Unlike the traditionally way with jQuery you do not have to add event listeners on CSS classes and select HTML elements by querying them.

AngularJS Bookmark App, list view.

read more

Comment [4]

RESTful API with Slim PHP and SQLite

Guido Krömer - 1216 days ago - Tags: , , ,

I needed a small RESTful API, primarily for playing around with AngularJS. The Slim Framework perfectly fits my needs. It is a PHP micro framework for writing a prototype or really small applications, without the overhead and the steep learning curve of a complete framework likes Yii, Symfony or Zend. At the same time, you get stuck at the Zend2 module configuration, you could have finished a working prototype with Slim.

I used SQLite as database for the API, because SQLite is configuration free it is perfect for prototyping. For keeping the focus, of this tutorial, at the Slim Framework itself, I skipped using any kind of ORM framework. Due the use of PDO any other SQL Database like MySQL could be used, too.

To gain some practical benefit from this tutorial the REST API going to be discussed in this tutorial manages bookmarks in a CRUD style. I hope this help me with my bookmark chaos among Linux, Windows, tablet and smart phone.

read more

Comment [20]

Search all config files for a specific term

Guido Krömer - 1221 days ago - Tags: , ,

A common problem, which of all ini file contains the part loading module XYZ?

The find command below searches all ini files in the /etc folder for the term "mysqli".

$ find /etc -name "*.ini" -printf "echo \"FILE: %f\"\ngrep \"mysqli\" %h/%f\n" | bash

Comment

Monitor file copy progress with find and watch

Guido Krömer - 1264 days ago - Tags: ,

Currently copying a large amount of files and wants to see if the job is still running?

watch

find lists all files in the folder and his sub-folders, wc -l count the lines from stdout and watch executes this every two seconds.

$ watch "find ./ -type f | wc -l"

Comment

MySQL rename a database with bash

Guido Krömer - 1300 days ago - Tags: ,

Renaming an MySQL table is straightforward with RENAME TABLE, but renaming a whole database with RENAME DATABASE has been removed in MySQL 5.1.23 because it could be dangerous.

Since I needed this functionality some lines of bash does nearly the same by creating a new database and renaming each table like this: RENAME TABLE old_database.my_table new_database.my_table.

#!/bin/bash
execute_sql () {
  echo "$1" | mysql
}

rename_database () {
  FROM=$1
  TO=$2

  execute_sql "CREATE DATABASE IF NOT EXISTS $TO"

  TABLES=`execute_sql "SHOW TABLES FROM $FROM"`

  for TABLE in $TABLES; do
    execute_sql "RENAME TABLE $FROM.$TABLE TO $TO.$TABLE"
  done;

  echo "Done, drop the old database ($FROM) if needed."
}

rename_database $1 $2

Comment

Merging files together with the PowerShell

Guido Krömer - 1313 days ago - Tags: , ,

After playing a round with TypeScript, I needed a way for merging all those single files together. Performing this task with Bash is quite simple. Under Windows there is no Bash, but a thing called PowerShell. The PowerShell has got a large amount of Bash like aliases, which makes the move from Bash to PowerShel really simple, as you can see in the script below.

$TARGET_FILE = "Full.ts"

rm $TARGET_FILE 

$FILES = ls -Recurse -Filter "*.ts"

foreach ($FILE in $FILES) {
  cat $file.FullName >> $TARGET_FILE 
}

tsc $TARGET_FILE

Comment

Embedding a single file from Gist

Guido Krömer - 1327 days ago - Tags: ,

It seems GitHub removed the embed single file "Button" from Gist. Maybe they just moved it where I am unable to find it :D . This is really annoying since I had to inspect old sources blog posts to figure out how the query, for embedding a single file, has to look like.

So it's just a personal/public note for me.

<script src="https://gist.github.com/%NUMBER_OF_GIST%.js?file=%FILE_NAME%"/>

Comment

Canvas Raycaster

Guido Krömer - 1339 days ago - Tags: , ,

When I was young I loved Games like Wolfenstein 3D, Nitemare 3D, Corridor 7: Alien Invasion and of course Doom or Duke Nukem 3D. Since Doom and Duke Nukem 3D using complex raycaster engines with a BSP for storing the map and evaluating which parts has to be drawn I'm going to try a Wolfenstein 3D like raycaster, which is very simple and can be implemented very easy in JavaScript using the canvas element for drawing. Raycaster with default settings on Opera

read more

Comment [3]

Splitting an Image in PHP

Guido Krömer - 1347 days ago - Tags:

Some simple tasks ends up into a simple script. After trying to create a comparison image, which consists out of some screenshot taken at the same resolution with different application settings, details for example, I finally got mad creating this composed screenshot with GIMP.

Vertically splitted view on Safari, Opera and Firefox.

read more

Comment [1]

Speeding up canvas drawing by scaling it with CSS3

Guido Krömer - 1356 days ago - Tags: , ,

Playing around with HTML5 Canvas is a lot of fun, but drawing is really slow and reminds me on GDI+ or Java 2D. The larger the resolution of the drawing canvas is the worse the performance becomes. Doing a little game using the whole browser given size on a Full HD monitor can even slow down on Chromium with a relatively fast CPU (AMD A8-3870). My idea was drawing the canvas in a smaller resolution, the half of the available for example, and scale it up. This is the same way a game console like XBOX 360 and PS3 rendering Full HD or even HD Ready which sometimes is not HD Ready indeed. Scaling elements can be done using CSS3 transform.

Read more if you wants to know about the performance impact by scaling a canvas up instead rendering on a larger one:
Canvas scaled rendering performance on Firefox, Opera and Chromium

read more

Comment

Mono, WebClient and "Invalid certificate received from server."

Guido Krömer - 1359 days ago - Tags: , ,

Playing around with ASP.net MVC on Mono and the Facebook API Login for Server-side Apps the System.Net.WebClient threw a TslException with the message: "Invalid certificate received from server. Error code: 0xffffffff800b010a". Invalid certificate received from server.

After that, I experimented importing all general certificates using mozroots.

$ mozroots --import --ask-remove

But the error still occurred so that importing the needed certificates directly by defining the URL using the certmgr solved the problem.

$ certmgr -ssl https://facebook.com

Comment

Some bash special variables you should never forget

Guido Krömer - 1361 days ago - Tags: ,

Since the main topic here in the last time was JavaScript here is a small foray into the world of bash special variables. Although there are a lot more special variables in bash this might be the most common in my opinion.

Getting all params

There are two variables containing all params, the first one is $* which represents all params as single quoted string and $@ where each word is quoted separate. If it's not clear what is going on there, let's have a look at the example below.

#!/bin/bash
ALL_ARGS_STAR=("$*")
echo STAR: ${ALL_ARGS_STAR[0]}
echo STAR: ${ALL_ARGS_STAR[1]}
ALL_ARGS_AT=("$@")
echo AT: ${ALL_ARGS_AT[0]}
echo AT: ${ALL_ARGS_AT[1]}

Accesing the variables ALL_ARGS_STAR and ALL_ARGS_AT using the array indexer should make the difference of $* and $@ more clear.

[caco@MacBook-Air ~]# ./params-list.sh Foo Bar 1337
STAR: Foo Bar 1337
STAR:
AT: Foo
AT: Bar

The other usefull ones

  • $0, which is the first param and holds the called script name including the path.
  • $# represents the number of arguments.
  • $1 .. $n access the param directly.
  • $$ gets the PID of the running script.

If you got confused by $0, $#, $1337 or $$ have a look at the self explaining script below:

#!/bin/bash
SCRIPT_NAME=`basename $0`
NUM_ARGS=$#
PID=$$
ARG_ONE=$1
ARG_TWO=$2
ALL_ARGS=$@

echo "Script name: $SCRIPT_NAME"
echo "Number of arguments: $NUM_ARGS"
echo "PID: $PID"
echo "Arg one: $ARG_ONE"
echo "Arg two: $ARG_TWO"

i=1
echo -e "Arg:\tVal:"
for ARG in $ALL_ARGS; do
  echo -e "$i\t$ARG"
  i=$(( $i + 1 ))
done

Here is the generated output:

[caco@MacBook-Air ~]# ./bash_params.sh Foo Bar 123
Script name: bash_params.sh
Number of arguments: 3
PID: 25052
Arg one: Foo
Arg two: Bar
Arg:    Val:
1   Foo
2   Bar
3   123

Comment

Node.js Chat client with jQuery Mobile

Guido Krömer - 1370 days ago - Tags: , ,

Welcome to the last part of my node.js chat tutorial, the topic of this part is the jQuery Mobile based client. The decision choosing jQuery Mobile has the two reasons, the UI I created for "WebSockets chat with Node.js " was really ugly because I'm not a designer, so I decided taking something which looks well out of the box. The second reasons was that jQuery Mobile seems to be well tested on different browsers even if they are not WebKit based. If you have not read the server side tutorial you maybe should read it before.

The jQuery Mobile chat client
This screen shots show how the chat client will look.

read more

Comment [3]

sprintf in JavaScript

Guido Krömer - 1376 days ago - Tags:

This is my implementation of a sprintf like function in JavaScript. It does not support any fancy number formatting and the replacement character (%) could be replaced, too. But sometimes it is just good enough.

String.prototype.sprintf = function () {
    var string = this;
    for (var i = 0; i < arguments.length; i++) {
        string = string.replace('%', arguments[i]);
    }
    return string;
}

My sprintf is a method of the string "class", so the call would be myString.spintf(1, 2, 3) and not like in c: sprintf(myVar, 1, 2, 3):

console.log('a: % b: % c: % d: %'.sprintf('A'));
console.log('a: % b: % c: % d: %'.sprintf('A', 'B'));
console.log('a: % b: % c: % d: %'.sprintf('A', 'B', 'C'));
console.log('a: % b: % c: % d: %'.sprintf('A', 'B', 'C', 'D'));

Here is the output produced by the code above:

$ node sprintf.js 
a: A b: % c: % d: %
a: A b: B c: % d: %
a: A b: B c: C d: %
a: A b: B c: C d: D

So have fun using and modifying it.

Comment

Serving static files with Node.js

Guido Krömer - 1388 days ago - Tags: ,

Node.js isn't built for serving static files, but during testing and development it's quite handy doing this without configuring an apache or whatever http server. No configuration file is needed just two lines of JavaScript to get this server running.

var MiniHttp = require('./MiniHttp.js').MiniHttp;
var server = new MiniHttp('/tmp/', 8000, null, 'foo.html');
read more

Comment

Use a QEMU qcow2 hard disk image with VirtualBox

Guido Krömer - 1411 days ago - Tags: , , ,

Last week I had to use an existing virtual machine on my local computer, which is a mac. The origin host machine was running Linux with KVM for visualization. The disk image format is qcow2, which is the default format in QEMU and not supported by VirtualBox. But the qemu-img tool can convert a qcow2 image to VirtualBox’s vdi format.

# qemu-img convert -O vdi vda.img vda.vdi

After converting the format, the new disk image, named “vda.vdi”, can be used with VirtualBox.

Comment

Creating a iso image with dd and Mac OS X

Guido Krömer - 1423 days ago - Tags: ,

Thank god, dd works on Mac OS X, too. Just login as root unmount the eventually automounted disc and use dd as usual.

[caco@MacBook-Air ~]# sudo -s
Password:
[root@MacBook-Air ~]# mount
/dev/disk1 on / (hfs, local, journaled)
devfs on /dev (devfs, local, nobrowse)
map -hosts on /net (autofs, nosuid, automounted, nobrowse)
map auto_home on /home (autofs, automounted, nobrowse)
/dev/disk2s0 on /Volumes/XP_PRO_SP3 (cd9660, local, nodev, nosuid, read-only, noowners)
[root@MacBook-Air ~]# umount /dev/disk2s0
[root@MacBook-Air ~]# dd if=/dev/disk2s0 of=Documents/MY_IMAGE.iso

Comment

Remove the phassprase from a ssh keyfile

Guido Krömer - 1424 days ago - Tags: ,

Just a command you maybe need sometimes when a shh keyfile with a phassprase becomes annoying. You will still need the actual phassprase(123456 in the example below) to remove it.

# ssh-keygen -p -P 123456 -N "" -f /home/user/.ssh/id_rsa

Comment

Prevent impatient users to click a link several times

Guido Krömer - 1440 days ago - Tags: ,

  • Fact 1: Users are very impatient.
  • Fact 2: Some pages have a really long request time.
  • Fact 3: Users try to improving the request time by clicking several times the same link.

This can be bypassed by disabling the links on a page when they got clicked.

$('.one_click_link').click(function() {
    $(this).click(function() {
        return false;
    });
})

This solution has one disadvantage, if the user decides to click another link and reverts his decision the already clicked links are still disabled. This is the reason I used the class "one_click_link" which will be assigned to "problematic" links, instead of giving all links on a page this availability using $('a')...

Comment

MongoDB: manipulating values if a "update" is not enough

Guido Krömer - 1444 days ago - Tags: , ,

Sometimes the MongoDB update function does not fit the requirements. Since JavaScript can be executed directly in the database (if it's not disabled) it can be used for more complex updates, for example. This happens without moving the data to the application and back to the database.

This small example does the same as: myColl.update({myValue: 1}, {$inc: {myValue: 1}})

db.myCollection.find({myValue: 1}).forEach(function(doc){ 
    var newValue = doc.myValue + 1;
    db.site.update({_id: doc._id}, {$set: {myValue: newValue}})  
})

This is a concrete example where tags have been saved as a comma separated string. Creating an index, for fast searching in the tags, on this field would not make sense. Converting this string field into an array of tags should solve the problem. This piece of code does this job for each site which has been tagged, for speeding up only the tags field and MongoDB's _id field gets queried from those documents.

db.sites.find({tags: { $exists: true }}, {tags: 1}).forEach(function(doc){ 
   db.site.update({_id: doc._id}, {$set: {tags : doc.tags.split(',')}})  
})

Comment

Merging files together

Guido Krömer - 1452 days ago - Tags: ,

Having a website CSS or JS separated into different files and sorted in sub folders is great for development, but ends up in many requests during live operation. Instead of using a “complex” script which merges those files together this one liner can do the job, too.

# find /path/to/css/* -type f -name *.css | xargs cat > /path/to/css/screen.css

Comment

Auto cleaning up your download folder with find

Guido Krömer - 1473 days ago - Tags: ,

Is your download folder messy, too? Why not let find tidy it up periodically?

# find ~/Downloads/* -ctime +7 -exec rm -rf {} \;

The command above let find search for files and folder which are older than 7 days and deletes them. But be carefully files in a folder will be deleted even if the file age is newer than 7 days if the folder is older than 7 days!

0 * * * * find ~/Downloads/* -ctime +7 -exec rm -rf {} \; > ~/Downloads/deleted.files

Running this as a cronjob will do the work once a hour, or how often you wants.

Comment

FTP upload via cURL

Guido Krömer - 1475 days ago - Tags: , , ,

Uploading a file to a ftp server with cURL is really easy:

# curl -u user:pass -T my_file ftp://example.com/target/path/

Or with a little potion of bash for uploading multiple file:

# for file in ./*; do curl -u user:pass -T "$file" ftp://example.com/target/path/; done;

Comment