RETS: Small Name, Big Possibilities

What is RETS? At first glance it sounds like something you may want to avoid, but for the professional Realtor and their web developer it’s like gold. It helps Realtors update property information on the MLS server, saving time and keeping listings current. It can all be done through PHP and a good database.

Currently my hometown has been getting a huge influx of people moving to our small community from the expensive big city. Our local Realtors our trying to capture these prospective clients as they move in, and one way they are doing this is by highlighting the properties they are selling through their website. However most of their websites are static and could use some updating, but they, like many others, are too busy with business to worry about website updates. They need to have their home and property listings pulled to the web automatically to save time and money.

This is where RETS can help.

Realtors who understand RETS love the idea, but many have not heard of it or what it can mean for them. Most of the time they input their property information on the MLS server and will continue to update this information through the sale of the house. Although, many times the Realtor will list the same property on their own website, but it will usually go unchanged for months and may even remain on their own site after it has been sold, only to cause confusion by potential buyers. RETS can alleviate the double input. Once the Realtor has added or updated their property information on the MLS server, their website can be updated without having to lift a finger, and it can all be done through PHP and a good database.

The History of RETS

RETS is a set of standards developed by the NAR (National Association of Realtors) in 1999 to access MLS information from a computer, based on a single set of standards and a common language. This gives the Realtor the ability to pull any information stored in an MLS server. There are many MLS service providers across the nation; now with this set of standards, the Realtor is able to pull information for their own purposes from any MLS in the nation – talk about open.

This is why it is so useful. Hundreds of thousands of homes may be for sale, and you could go to one site to see current information on any house or property in any state that is for sale, much like homestore.com. Sites like the one I just mentioned can use this technology to constantly update their databases of property for sale and keep content fresh for property across the nation.

The standards used are XML based, which makes them easy to read and parse by a program to store in a database for later use.

RETS Defined: Real Estate Transaction Specification 

The Real Estate Transaction Specification is a document that defines a series of computer interactions called “transactions” and serves as the definitive reference source for developers who implement them. It establishes the parameters for each specified transaction, both upload and download, as well as the expected behaviors and results that compliant hosts and clients must be able to manage, including error codes.

For more information go to http://www.rets-wg.org/faq.html.


{mospagebreak title=How it Works}

You first have to develop a client. A client is a program that connects to the RETS server and extracts certain information and displays or places what it pulled into a database. There are many pre-made clients available, some even written in PHP, but I found that they do not work for every RETS server straight out of the box and require a bit of updating to work with a specific RETS server, which is not an easy task when the code does not have sufficient comments or documentation. That is why I suggest building your own; usually one of the best ways to learn anything is doing it on your own. The Wright Brothers would have never built their first plane if they were waiting to update someone else’s plane. Let’s look at what the client program will have to do.

Most of the RETS servers implement Digest Authentication. This is much more secure than your usual Basic Authentication, which makes it a little more complicated to program. Digest uses MD5 to encrypt usernames and passwords and other variables so that information is not sent over the Internet. We will get into the login process a little bit later in this article.

Once you log in, you will receive a session id which you will need to save so you can log back in to request information. RETS servers have many options to pull information out; you can search by price, amenities, location, school district, Realtor office and basically anything else that is a field in the RETS server.

This is how a common session might look.

The Common Session 

Program Step 1
Logs in and retrieves the session_id and then logs out

Program Step 2
Logs in using the session_id and requests a list of property id numbers and then logs out

Program Step 3
Logs in using the session_id and requests detailed information on specific id number and then logs out

Program Step 4
repeat step 3 for all property ids in the database

Program Step 5
session completed.

Once you have a successful login and request information, the RETS server will send a page back with all the information you requested. The information will be in the same order that you sent the request, so if you asked for bedrooms, bathrooms and pool, it would come back in that same order. The content will be in a format that looks like the below sample.

Sample Data 

HTTP/1.1 200 OK (This lets us know everything was fine retrieving from this page)
Server: Microsoft-IIS/5.0 (Type of server the RETS server is)
Date: Thu, 08 Jul 2004 23:05:33 GMT (The date and time of the transaction)
X-Powered-By: ASP.NET (What type of programming language is used on the RETS server)
Cache-Control: private
RETS-Version: RETS/1.0 (The standards version the RETS server complies with)
Content-Type: text/plain (Format of the information below)
Cache-control: private
 
<RETS ReplyCode=”0″ ReplyText=”Operation Successful.” > (Everything is ok)
<COUNT Records=”4″ /> (Number of records were found)
<DELIMITER value=”09″ />
<COLUMNS> ListingPrice MLNumber Status </COLUMNS> (How the data is formatted)
<DATA> 198500 13603 A </DATA> (The data)
<DATA> 93000 13880 A </DATA>
<DATA> 318000 14029 P </DATA>
<DATA> 275000 14056 P </DATA>
</RETS>

You can see from the above sample data that it is in an XML format which makes it really easy to parse through and place into a database.  This is similar to RSS news feeds, which makes it very easy to place news into a database.  You may have tried to parse MLS data off of a website before  – it is nearly impossible, this is why this is so great; you get the data in a very easy to parse format and can worry about bigger things like how to tell your co-worker that a water cooled computer does not mean pouring water on the processor every few hours.

The system we built in our office checks the RETS server every hour for any updates and adds and removes properties based on the information that the server spits back. We have automated it through a cron job and we can just sit back and let the computer do all the work, day and night.

{mospagebreak title=The Basic Login in Detail}

There is little information available for programming a client in PHP. There is a large document about the standard and how things should work, but an actual step by step of how to build a client in PHP does not exist. When I started on the project at our offices, I found very little help. There were some people who had successfully built a client, but they did not want to share their findings or even offer a little direction. This can happen when new technologies come out; people sometimes try to hoard information to make themselves look better instead of advancing the possibilities of a technology. Where would PHP be today if it was not open? Hopefully this will get you started and over one of the biggest hurdles in developing your client, the login. Once I got past this, everything came together.

The login process is not too difficult, but is a little tricky because we will be using MD5 and digest authentication. This is a very secure way of logging, and it is understandable that most RETS servers will use this type over Basic Authentication. Some older servers still use Basic Authentication, but it is rare.

Now let’s look quickly at MD5 and digest authentication. MD5 is a way to verify data integrity. It takes a string of text and returns a 128bit “fingerprint”, 32 characters of letters and numbers. The reason for this is that the actual string is never transmitted over the Internet, and it is very difficult if not impossible to decrypt this fingerprint. When the computer you are sending to receives the request of authentication, it will MD5 a similar string on it and compare the one you sent over with the one it made. For example if you sent over a password “cat” that was encrypted using MD5, which may look like this “1f3870be274f6c49b3e31a0c6728957f” to another computer, it would take the password that it had in its database for you and encode it using MD5, and if your string matched what the computer had, it would allow you access. MD5 can be used for many things, even checking if two sentences are identical, basically anything that needs to be compared to each other for equality.

The login script consists of MD5 variables and header information sent over the Internet using fsocket to a specified port for authentication. Once there is an approved login, the MLS server will display a session which you can use for data retrieval.

{mospagebreak title=The Login Script}

With that being said let’s look at the first part of the login script (the full code is included).

<code>
$url = “rets.somemls.com”;  
</code>
This is the URL we will use through out a session.  The MLS office will have to provide this to you, it is usually not public information.

<code>
$fp = fsockopen($url,80,$errno,$errstr,30);
if (!$fp) {

      echo “$errstr ($errno)<br />n”;
} else {   
</code>

This is a standard fsocket connection procedure. Notice that it is going to port 80 - some MLS systems might use other ports to pull RETS information from. The number 30 is the time out limit. If there is an error the above code will let us know. For more information on fsocket go to the  manual http://us4.php.net/manual/en/function.fsockopen.php

<code>
$username = “retsuser02″;
$MLS_county = “LAMLS”;
$password = “getinfo”
$A1 = “$username:$MLS_county:$password”;
</code>

You will use your username and password here that the MLS office gives you. Also you will notice that between the username and password is a variable we called MLS_county, this will be a variable that the MLS office must give you. All counties have a different code and if the correct one is not used, there will be an error when you MD5 this variable. This specific arrangement of these variables may be different from MLS office to MLS office; there is a standard but it does vary slightly usually for security purposes and the MLS service provider will make these differences known.

<code>
$A2 = “GET:/IMPV/rets/rets_login.asp”;
</code>

This is the page that will be used to authenticate you as a user of the system. This page will need to be given to you by the MLS office.

<code>
$auth_nonce = “adee8373e2588003c1746f58c89b30cd”;
</code>

Nonce is a string of randomly generated letters and numbers or current time generated by the server you are contacting. The MLS system I currently use does not change this variable and is a mix of numbers and letters. The way you get the nonce is to go to the page without logging in and review the headers and pull out the variable “nonce= adee8373e2588003c1746f58c89b30cd”. You may need to add a section of code that goes to the page and parses out the nonce and opaque variables which we will use further in the code.

<code>
$raw_digest = md5($A1).”:”.$auth_nonce.”:”.md5($A2);
</code>

Now let’s MD5 our variables. No matter how long the variable you MD5, it will encrypt it into a 32 character variable. You will notice that the $A1 and $A2 variables are MD5ed and the nonce is not. This is just the way to make things even more complicated for someone trying to decrypt this string.

<code>
$resp = md5( $raw_digest );
</code>

The last step is to MD5 the variable we just MD5ed. This may vary from each MLS office, but it does provide another step of security. This variable must match an MD5 variable the MLS service provider generates on its end; if it doesn’t then there will be no authentication. And again the benefit of using MD5 is that no obvious information is being sent over the Internet.

{mospagebreak title=Authentication}

The lines below will be the header we create and send over for authentication. These are the usual minimum requirements but again may be slightly different from MLS server to MLS server. You may want to take some time before trying the authentication process to locate and check the requirements of your local office; they are usually posted online.

<code>
$out2 = “GET /IMPV/rets/rets_login.asprn”;
</code>

The page we use for authentication. Notice that it differs slightly from our $A2 variable.

<code> 
$out2 .= “Authorization: Digest username=”$username “, realm=”$MLS_county “,
nonce=”$auth_nonce “, opaque=”$auth_opaque”, uri=”/IMPV/rets/rets_login.asp”, response=”$resp” rn”;                
</code>

This is the actual digest string that brings everything together for authentication. Some of the variables we created above are placed here and our MD5ed variable $resp is in the response area at the end.

You will notice that there is a variable called opaque. This variable will appear when you get the nonce variable from the header and looks similar to the nonce. You can pull it the exact same way that you pull the nonce. Again my specific MLS server does not change this variable, but others might.

<code>
$out2 .= “Host: 63.171.248.226rn”;
</code>

The server that you will post this code to and run the program from.

<code>
$out2 .= “Accept: */*rn”;
</code>

The type of response you will accept. Some MLS server may require you to specify the type which may be text or XML. The above is set to anything and everything.

<code>
$out2 .=”User-Agent: MY_RETS_CLIENT/1.0rn”;
</code>

This is the name of your client; this will need to be registered with the MLS office. This is to prevent any program from trying to gain access without properly registering with the MLS office. You may be familiar with this variable which usually holds information about your browser like “Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-us)”. We are basically spoofing the header; if it’s not set, it would probably say “PHP 4.2.3″ or whatever version your PHP is on the server.

<code>
$out2 .=”RETS-Request-ID: 571rn”;
</code>

This is used by your program to keep track of requests. Every request could have a unique number assigned to it for logging or record keeping. This is usually an optional variable.


<code>
$out2 .= “Connection: Closernrn”;
</code>
   

Finally close the connection.

<code>
fputs($fp, $out2); 
</code>

Now we connect to our fsocket connection and send over the header we just made. The header is a GET statement and not a POST, information that you may not need but is good to know how things are being sent.

<code>
while (!feof($fp)) { 
   
        $grabo = fgets($fp, 128); 

      preg_match(“/RETS-Session-ID(……….)/”, $grabo, $set_cookie);
     if($set_cookie[0] != ”){
      $set_cookiea = $set_cookie[0];
      }
      }  

fclose($fp); 
}
</code>

This bit of code retrieves the server response once we send our authentication. If we are successful, it will try to find a variable called RETS-Session-ID. If we get this variable, then it means that we have successfully logged into the server and can process requests for property using that session id.

{mospagebreak title=Benefits of RETS}

The log in process is not a difficult one to understand, but there are a lot of little steps that must be absolutely correct for us to be authenticated. You will have to check with the MLS office to get the exact parameters to MD5. They sometimes vary from office to office, but most MLS systems will post this information to their websites. Even though there is a standard, there are slight variations from office to office, and you will need to check before you get started.

Once you do log in successfully and receive your session variable, you will be able to freely request information, perform searches for properties and even download raw images to place into a database. Of course the obvious benefit for using RETS is the ability to get current information and place it into a database without having to retype or parse some very tricky html page. Using RETS can open the door to many business opportunities. With the knowledge you have in connecting and pulling information from other computers, it may open doors later on for similar systems.

One quick note on this whole login process - what we have just looked at is a digest method of authentication. This type of authenticating is used by other types of web services in different areas and is not specific to RETS. The code is not proprietary to RETS but a common way to gain access to a server, and RETS uses XML to send back data, so that too is not proprietary. This same code, with some modifications, can be used for other login scripts. This is another reason why trying something on your own will usually benefit you and future programs.

There are a few RETS clients out there, but building one from scratch will give you a custom application that you can change and upgrade yourself at any time. Things you learn from one project will usually carry over to another one - at least that’s what I have found. One thing that I wouldn’t suggest you do yourself is put up crown molding in a room - your thumbs and brain will thank you.

{mospagebreak title=Final Thoughts}

The standard is good and is also in its starting stages. The server I log into uses the RETS standard 1.0 and is currently in the process of going to 1.5. Most of these service providers offer backward compatibility, which is good for those whose clients won’t pay for an upgraded product. It will be interesting to watch as RETS grows and how it will be used in the future.

The important thing about the RETS standard is the use of a standard to share information from one computer to another. This is a great example of a standard that has been accepted and actually used across a large area. The power of the standard can be seen by looking at homestore.com, which allows a user to view houses and property for sale from all over the US. Imagine if other areas of industry could develop standards to share information freely. Currently the RETS standard is not international but specific to the US, but there are rumors of getting the RETS standard into the international arena. I know to some this may be a scary thought, and this is one reason why the standard has not gone international. Some see it as stripping away other’s culture for western ideals. I don’t know about that, but having everything the same does leave little room for individuality.

Conclusion

Now as a web developer you can see the potential to help your local Realtors create better and more exciting websites using RETS. This is why it’s like gold, the Realtor benefits from the automatic updates to their website, and the web developer benefits from building a program to interface with RETS.

The nice thing about the whole RETS standard is that it is open and continually being developed to make it better and easier to share information. It is not proprietary and does not require any one piece of software, so that means anyone can create a client and have Realtors use it. In the example here we used fsockets, but you could develop it using other tools like curl. Most of the clients available use Java, but you can see it is open and defiantly not proprietary. You can also use your client over and over for any Realtor that would like to use it with small modifications. Also the things you learned in creating your client will work for other projects.

The main area for information on the RETS standard is http://www.rets-wg.org/. Another good source of information and pre-made open source RETS clients is http://www.crt.realtors.org/. Documentation http://www.rets.org/docs/.

To gain access to RETS, you will have to get a username and password from the RETS or MLS provider for your area, but it is a fairly painless process.

Google+ Comments

Google+ Comments