ME Tech

Quick and dirty resque retry all failed tasks

Matt — Thu, 02 Feb 2012 20:37:56 +0000

If you have a lot of failed tasks staked in resque, you can retry them all in one shot, but it’s the dirty way.

>> (Resque::Failure.count-1).downto(0).each { |i| Resque::Failure.requeue(i) }

Dirty because it does not dequeue task when there done.
You should then remove the after having checked they really succeed.

Now you can find resque contrib for this kind of issues (retry count and so on…).

Array in ruby short tutorial: ruby way for ruby noobs

Matt — Wed, 18 Jan 2012 09:44:46 +0000

Arrays in ruby the ruby way (with code blocks).

Instanciate an array from words

>> els = %w(one two three four five)
=> ["one", "two", "three", "four", "five"]

Iterate over the array

>> els.each { |el| puts el }
one
two
three
four
five

Build an array invoking a code block on each element

>> els.map { |el| el.reverse }
=> ["eno", "owt", "eerht", "ruof", "evif"]

Return only elements matching a criteria

>> els.select { |el| el =~ /t/ }
=> ["two", "three"]

Reject all elements matching a criteria

>> els.reject { |el| el =~ /t/ }
=> ["one", "four", "five"]

Combine all elements applying a code block

>> els.inject('my elements are') { |str, el| str += ' ' + el }
=> "my elements are one two three four five"

tilde on mac (~) #tips #microblog

Matt — Thu, 05 Jan 2012 16:08:19 +0000

To obtain the tilde character on mac,

Press Option + n and you get ~

Press Option + n and then a and you get ã

weekly digest: #scalability, page load performance, google prediction api, httperf

Matt — Mon, 14 Nov 2011 21:18:55 +0000

List of Known Scalable Architecture Templates

http://srinathsview.blogspot.com/2011/10/list-of-known-scalable-architecture.html

You only control 1/3 of your Page Load Performance!

http://blog.dynatrace.com/2011/11/08/you-only-control-one-thrid-of-your-page-load-performance/

Google Prediction API

Machine learning cloud by google.

http://code.google.com/intl/fr-FR/apis/predict/

How to replay live traffic with httperf

Tips to test uris from your nginx logs against your server.

http://schlinkify.org/post/19743846/how-to-replay-live-traffic-with-httperf

NoSQL Europe : Bases de données graphe et Neo4j

French intro to the Neo4j graph database.

http://blog.xebia.fr/2010/05/03/nosql-europe-bases-de-donnees-graphe-et-neo4j/

Architecture and infrastructure scalability paper and resources

Matt — Fri, 21 Oct 2011 12:24:07 +0000

Architecture & Infra #scalability is an automatic paper compiling best resources about scalability and architecture.

Sources are taken from a twitter list and a google reader rss folder.

Twitter list

@mathieuel/architecture this is tech people i enjoy follow on this topic.

RSS List

Architecture from mathieue contains all the rss feeds i’ve subscribed on this theme.

All Things Distributed Werner Vogels’ weblog on building scalable and robust distributed systems.
Amazon Web Services Blog Amazon Web Services, Products, Tools, and Developer Information…
about:performance Application, performance, scalablity and architecture.
Xebia nosql category
Flickr tech blog
Decrypt mainly in french
Digg Technology Blog
Facebook developers
High Scalablity my number one destination site
Kitchen Soap Thoughts on capacity planning and web operations.
Nati Shalom’s Blog Thoughts on Scalability, NoSQL, Big Data, DevOps, Cloud, PaaS
ProductionScale Kent Langley’s Blog on Cloud Computing and Business Architecture
Scalable web architectures Building reliable, high performance, highly available clusters
Twitter Engineering
igvita.com A goal is a dream with a deadline

Of course, if you have other resources, even your own twitter account, feel free to send it to me !

Amazon ec2 tutorial – get started quickly

Matt — Mon, 19 Sep 2011 19:54:45 +0000

The goal of this amazon ec2 tutorial is to provide the simplest way to have a virtual machine server running on amazon vms environment. It runs a ubuntu server on the smallest instance available.

On the amazon console website

Create a X509 certificate on amazon aws console and download AT THE SAME TIME certificate and private key.
Private key

/tmp/pk-WSXXXXXXXXXXXXXXXXXXXXXXXX.pem

Certificate

/tmp/cert-WSXXXXXXXXXXXXXXXXXXXXXXX.pem

Install ec2 client on your server

bob@server~: mkdir ~/.ec2
bob@server~: wget http://xxxxxx/ec2-api-tools.zip # get the url from the official site
bob@server~: unzip ec2-api-tools.zip
bob@server~: mv ec2-api-tools/* .ec2 && rmdir ec2-api-tools
bob@server~: cd .ec2/
bob@server~: mv /tmp/pk-WSXXXXXXXXXXXXXXXXXXXXXXXX.pem .
bob@server~: mv /tmp/cert-WSXXXXXXXXXXXXXXXXXXXXXXX.pem .
bob@server~: sudo apt-get install openjdk-6-jre
bob@server~: vi ~/.bashrc

Put this on your ~/.bashrc

export EC2_HOME=~/.ec2
export PATH=$PATH:$EC2_HOME/binexport
export EC2_PRIVATE_KEY=$EC2_HOME/pk-WSXXXXXXXXXXXXXXXX.pem
export EC2_CERT=$EC2_HOME/cert-WSXXXXXXXXXXXXXXX.pem
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/# if your want to work in european datacenter, set it by default# you can ignore this conf
export EC2_URL=https://ec2.eu-west-1.amazonaws.com

Test the client: it should be ok now

bob@server~: ec2-version
1.4.4.2 2011-07-15
bob@server~: ec2-describe-images -o self -o amazon
IMAGE ami-2e53785a ec2-public-windows-image............

it’s ok !

Push your key pair and run your VM

bob@server~: ec2-import-keypair workstation-melie -f /home/bob/.ssh/id_rsa.pub
bob@server~: ec2-run-instances --instance-type t1.micro -k workstation-melie ami-359ea941
bob@server~: ec2-describe-instances

If you can see your instance,

Log in the VM

ssh ubuntu@ec2-46-137-143-104.eu-west-1.compute.amazonaws.comubuntu@ip-10-227-177-18:~$
# you should be in your vm !
ubuntu@ip-10-227-177-18:~$ sudo su
root@ip-10-227-177-18:/home/ubuntu# halt

Kill your VM.

bob@server~: ec2-terminate-instances i-7d5eba34

Handy commands

Listing your available keypairs

bob@server~: ec2-describe-keypairs

Available regions id and api endpoint.

bob@server~: ec2-describe-regions
REGION eu-west-1 ec2.eu-west-1.amazonaws.com
REGION us-east-1 ec2.us-east-1.amazonaws.com
REGION ap-northeast-1 ec2.ap-northeast-1.amazonaws.com
REGION us-west-1 ec2.us-west-1.amazonaws.com
REGION ap-southeast-1 ec2.ap-southeast-1.amazonaws.com

improved gnu screen life with 9 lines bashrc

Matt — Sun, 21 Aug 2011 12:50:45 +0000

alias sl='screen -list | sort -t "." -k2'

alias s='screen -RRd'

_melie-screen()
{
local cur=${COMP_WORDS[COMP_CWORD]}
COMPREPLY=( $(compgen -W '$(screen -list | cut -f 2 | cut -d "." -f 2 | grep -v "There" | head -n-2 )' -- $cur) )
}
complete -F _melie-screen s

The super short s alias open OR reopen an existing screen with the supplied name.
s have completion enable which is a such power feature in shell in 6 line of bash.

top by process name

Matt — Tue, 16 Aug 2011 21:52:48 +0000

Monitor all process named apache2.
We user here pgrep. prep handle regexp.

top -p $(pgrep apache2 | xargs echo | sed -e 's/ /, /g')

note: it seems top -p have 20 pids input limit . Have a better solution ?
pidstats takes process name, but have other features…

linux virtualbox vm on mac without visual interface

Matt — Thu, 04 Aug 2011 20:49:38 +0000

Sometines it’s handy to have a linux server vm running on your mac laptop / desktop but you want access it only like a server cause virtual box user interface could be annoying and useless here.

1 Set up or add a network interface in nat mode and add a nat rule from a local port (2222) to vm ssh port

check ssh service is running in your vm !

2 edit a helper bash script to launch the vm in headless mode (only in a term)

#!/bin/bash
# vmmacbook is the name of my vm

VBoxHeadless -startvm vmmacbook

3 stop…

#!/bin/bash
ssh mathieu@127.0.0.1 -p 2222 sudo halt -n

Now its painless to start and stop the vm. You can user it like a remote server

#microblog Data Visualisation Shows London’s Twitter And Flickr Traffic |…

Matt — Sat, 23 Jul 2011 16:33:18 +0000

Data Visualisation Shows London’s Twitter And Flickr Traffic | Londonist