Thinking Sphinx for Mighty tables

Recently i was assigned to integrate thinking sphinx in a messaging application and the messaging table was having more than 20 Million records.

Though i have configured thinking sphinx in different project so the estimation was that it will take some couple of hours to add it on server but this time i was wrong Because ...

In thinking sphinx the query we used to add delta filed is some thing like this

alter table table_name add column delta boolean default true

But when ever you run the rake command to build the indexes it will first execute the following query

update table_name set detla = false where delta = true


Wow what a beautiful query as in my case this query takes around 4 hours to complete and during this time whole table was locked and server remains down.

why this happens ?

Some projects nature is to introduce search in very initial level or with very little number of records in table so the second query runs very fast and after its completion indexing process starts. As the all db records are being getting qualified for where clause in second query and its going to update all records again so table gets lock.

The solution for this problem is to alter all records delta value back to false first in chunks, then run the rake task to index all records and then after wards index your records.

some thing like this

TableModel.find_in_batches do |records|
  TableModel.update_all {:delta => true}, ["id in (?)", records.map(&:id)]
end

and then afterwards running

rake thinking_sphinx:index

cheer :)

Local Server to Public

Most of time we need to communicate public ip for our system so external/3rd party services can communicate with our local running server. One of the solution can be requesting your ISP or Network service provider to redirect a particular port to your local machine, this is on old way which i use to develop my first facebook application.

But now thanks to localtunnel (though have some problem) is a good one to use a public domain at run time so your local server is accessible by public.

Very simple steps https://github.com/progrium/localtunnel described here.

just install them gem of localtunnel and run the command  localtunnel -k ~/.ssh/id_rsa.pub 8080 for first time and afterwards localtunnel 8080 in response it will give you like this Port 8080 is now publicly accessible from http://8bv2.localtunnel.com ...


That's it show your local server to public and make them happy by looking traces of the crashes ;)

Redis Memory Issue with Resque

Resque maintains its information/traces/jobs in Redis, a very pupular and fast in-memory key value storage.

After shifting to heroku the redis memory starts getting increase day by day and the temporary solution was to keep adding more powerful redis add-ons which was a poor approach and expensive one.

So decided to flush the old memory as there is no need to keep the old processed jobs data by resque. Thanks to Heroku's support time that they put the right direction for flush all of memory.


 uri = URI.parse(ENV["REDISTOGO_URL"])

 redis = Redis.new(:host => uri.host, :port => uri.port, :password => uri.password)

 redis.flushall

It will clear all your redis contents and if you look to resuque web console all counters and queues jobs has been reset and going to start from their seed values.

:)

(ActiveRecord::RecordNotUnique) "PGError: ERROR: duplicate key value violates unique constraint

Today when pushed the code on production a strange notification get starts mailing that record is not unique ... the primary key is being getting duplicate which is supposed to be unique and was set to auto-increment.

Pressure keep rising as the site was live and the delete functionality was getting crash here investigation started as.

1. Created a record from console and it get saved perfectly and get the id 225.
2. Check the last record of the culprit table and the id was 983
3. Here it clicks that wrapper of active model was introduced later and the table was being getting dumped through sql.
4. Take little help from google to check how postgres handles primary key and it comes to know that postgres maintains Sequence structure for primary keys and just keep single filed ++ for the new records unique id. 
5. So simply update the relevant tables primary key fields sequencer to the max id of the table and it starts getting smooth and no more Record Not Unique exception raised.

And this is how a strange issue of primary get resolved and came to know that the Rocket Science of DB's primary keys.

Cheer :)

HTTP Status Code usage

Status code specially using in Ajax calls.

Using http status code is now getting in practice and should be used to take advantages of these codes instead of handling them manually.

Making it simple by categorizing the response either success or failure where there responding codes are 200 and 400.

In context of rails there can be two ways to handle the response.

• First one the bad one

render :text => "Its working" and return

Now on rendering page we need to put string comparison to find out that either its success or failure response.

• Second one the good one

render :text => "Intelligent Text", :status => 200 and return 

Now here we don't need to make any comparison of responded text the status code will automatically map to relevant response.

like 

ajax:success( // do success stuff)

ajax:failure(  // do failure stuff)

That's it the relevant function will be invoked on basis of relevant status code.

Good to use status code.

:)

Rails 3.1.3 Assets Bug

A week ago working with a project and got a feel that my forms are being getting post more than one with random behavior, firstly i got a doubt on code bug but all was normal and then while debugging css i just saw in console that whole css has been loaded twice so this gives me a hint to the problem …

then here i start taking help of Sir Google and in one of blog i found a link that application.css or application.js use to load all files in assets folder and its getting load both compiled and non-compiled assets.

So the simple solution was to define assets for development environment by config.assets.prefix = “/devassets”

and this gives me a really relief as i have spent a lot of time in wondering the reason of forms getting multiple time posts even i tried to down grade rails to get rid of this problem but caught in more problems.

Resque Web Console

Resque a perfect queuing system up till now and well documented but i was unable to get that how to configure it with in application instead of running it separately …

As Resque console is sanitara app so we can mount it in rails 3 by just adding it in routes file

mount Resque::Server, :at=> “/resque”

and to add http authentication just add configuration file in initializers with following code

Resque::Server.use(Rack::Auth::Basic) do |user, password|

password == “password”

end

That’s it you are good to go with resque web console with your application server.

:)