I just updated my, and my R.R. teams, website Trendercast. I made it far more responsive with:
an OAuth sign-in process that won’t navigate away from your page (wont interrupt podcast playing)
a submission notification whenever new RSS feeds are entered (they take a while to gather and process)
Threaded RSS feed updates!!!
This is my first time trying out threads and I’m very happy with it.
I experimented in IRB like a good developer and found that calling the join method on a thread will have the “program” wait for the thread to finish before resuming. This is one feature I wanted to use in Rails … but guess what… using thread.join in Rails gridlocks the application. Thankfully I found a way around that. Here’s basically what the code looks like.
module FeedGenerator
def self.gen_feed(url, view_env)
# fetch record for RSS feed
feed = Feed.where(url: url).take(1).first
# If the feed is older than 4 hours we check to see if a new podcast has been posted
# but we don't want to delay the user's page loading so lets just start a thread in the
# background to perform fetching and DB updating
threads[url] = Thread.new { perform(url) } unless updated?(feed)
# if no record exists
if feed.nil?
# have the current process wait until data is finished being retrieved
sleep 3 while threads[url].alive?
# Allow the garbage collector to remove dead threads
threads[url] = nil
# We have to change the query so Rails won't use the previously cached result
# Grab the newly made feed
feed = Feed.where(url: url).where("created_at > ?", Time.now - 20.minutes).take(1).first
end
FeedPresenter.new(feed, view_env)
end
def self.threads
@threads ||= {}
end
def self.perform(url)
# grab RSS feed data and save to DB
end
def self.updated?(feed)
if feed
if feed.updated_at.between?(Time.now.ago(4.hours), Time.now)
return true
end
end
false
end
end
The server runs great and the experience is everything I had hoped it would be. My next step is to write a sleeper agent in JavaScript when there is an update to be checked for and dynamically insert it in the page once it’s available.
I might need to consider other users starting a thread for the same url at the same time. Perhaps I should rewrite the one line to:
# if feed is not updated and thread isn't alive (or exists)
if !updated?(feed) and !threads[url].try{|t| !t.alive? }
threads[url] = Thread.new { perform(url) }
end
There are things I don’t know… like if there are separate Ruby processes for every user or not. That would change what I would need to look for in threading.
Thanks Aston! It’s very informative. It doesn’t cover my curiosity in how Rails works. I have a general idea about Rack have a per route listener (thread)… but I’ve only imagined about how it might work.
There are many things I wonder about. Like in Minitest I would want to do things in a test file with just describe and it blocks. In Rails I’ve found tests to be very strict about what you can do based on what folder you’re in and a basic need for inheritance to add abilities.
There are “some” people out there who know these things and if they happen to come by and speak up I’m most delighted to learn. Otherwise I’d need to invest a lot of time studying source code. I’m not against that, I’ll get around to that when I need to. I’d rather have connections and dialog than be a loner who’s read the source to the whole world.
Yeah I totally hear you… hopefully when we get the new look a few new people will look in
Not sure if you are keeping a eye on the Elixir Forum but José Valim kindly created a section on the official Elixir lang site for us! I’m hoping the powers that be in Ruby world might do the same
Here’s hoping anyway
(Sorry to go off-topic - please feel free to delete or move to a separate thread)
@kofno Yeah, I do know Puma is thread oriented and Heroku is promoting it over Unicorn now.
On the topic of threads this showed up on my Twitter feed: Ruby Threads and ActiveRecord Connections and it was written a week ago. Good insight on database connection/pooling with threading.
And from what I hear Celluloid is the best system for handling threads in Ruby (even better in JRuby) and Sidekiq is the best threaded background system which has become mature enough to graduate away from Celluloid and they’ve written it now with pure Ruby threads. That is pretty much the extent of my knowledge on threads.
That and Ruby’s “gil” is problematic for threads except with cases of IO.
Celluloid use Fibers to organize his “tasks” flollowing the actor model into a single thread. But you can also use multiple thread each with there own pool of fibers (drived by Celluloid, of course). Be careful with JRuby : Fibers are real threads on this platform and you can reach the limit of your system. For MRI, it’s not a problem because fibers are not system threads (and should not be).
My threads are still working fine on the server. But they seem to have introduced an issue with DB access ActiveRecord::ConnectionTimeoutError. I need to somehow allow more access to the DB when threads start.
Spinning up a bunch of threads by checking all of the podcasts gets a bunch of threads reading for changes and writing updates. This can have some of the podcasts to seem to not load in the browser… probably the ones not in cache.
# if feed is not updated and thread isn't alive (or exists)
if !updated?(feed) and !threads[url].try{|t| !t.alive? }
threads[url] = Thread.new { perform(url) }
end
def perform(url)
ActiveRecord::Base.connection_pool.with_connection do
# ...
end
end
I just wrapped the one place I was calling Thread.new with the ActiveRecord::Base.connection_pool.with_connection and the site no longer comes to a halt.
