Create application based on Website

Issue

I searched and tried a lot to develop an application which uses the content of a Website. I just saw the StackExchange app, which looks like I want to develop my application. The difference between web and application is here:

Browser:

App:

As you can see, there are some differences between the Browser and the App. I hope somebody knows how to create an app like that, because after hours of searching I just found the solution of using a simple WebView (which is just a 1:1 like the browser) or to use Javascript in the app to remove some content (which is actually a bit buggy...).

To repeat: the point is, I want to get the content of a website (on start of the app) and to put it inside my application.

Cheers.

Solution

What you want to do is to scrape the websites in question by getting their html code and sorting it using some form of logic - I recomend xPath for this. then you can implement this data into some nice native interface.

You need however to be very aware that the data you get is not allways formated the way you want so all of your algorithems have to be very flexible.

the proccess can be cut into steps like this

1. retrive data from website (DefaultHttpClient and AsyncTask)
2. analyse and retrive relevant data (your relevant algorithm)
3. show data to user (Your interface implementation)

UPDATE

Bellow is some example code to fetch some data of a website it implements html-cleaner libary and you will need to implement this in your project.

class GetStationsClass extends AsyncTask<String, String, String> {

    @Override
    protected String doInBackground(String... params) {
        HttpClient httpclient = new DefaultHttpClient();
        httpclient.getParams().setParameter(CoreProtocolPNames.PROTOCOL_VERSION, HttpVersion.HTTP_1_1);
        httpclient.getParams().setParameter(CoreProtocolPNames.HTTP_ELEMENT_CHARSET, "iso-8859-1");
        HttpPost httppost = new HttpPost("http://ntlive.dk/rt/route?id=786");
        httppost.setHeader("Accept-Charset", "iso-8859-1, unicode-1-1;q=0.8");
        try {

            // Add your data
            List<NameValuePair> nameValuePairs = new ArrayList<NameValuePair>(3);
            httppost.setEntity(new UrlEncodedFormEntity(nameValuePairs, "utf-8"));

            // Execute HTTP Post Request
            HttpResponse response = httpclient.execute(httppost);
            int status = response.getStatusLine().getStatusCode();
            String data = "";
            if (status != HttpStatus.SC_OK) {
                ByteArrayOutputStream ostream = new ByteArrayOutputStream();
                response.getEntity().writeTo(ostream);
                data = ostream.toString();
            } else {
                BufferedReader reader = new BufferedReader(new InputStreamReader(response.getEntity().getContent(),
                        "iso-8859-1"));

                String line = null;
                while ((line = reader.readLine()) != null) {
                    data += line;
                }
                XPath xpath = XPathFactory.newInstance().newXPath();
                try {
                    Document document = readDocument(data);
                    NodeList nodes = (NodeList) xpath.evaluate("//*[@id=\"container\"]/ul/li", document,
                            XPathConstants.NODESET);
                    for (int i = 0; i < nodes.getLength(); i++) {
                        Node thisNode = nodes.item(i);
                        Log.v("",thisNode.getTextContent().trim);
                    }
                } catch (XPathExpressionException e) {
                    e.printStackTrace();
                }
            }

        } catch (Exception e) {
            e.printStackTrace();
        }

        return null;
    }

    @Override
    protected void onPostExecute(String result) {
        super.onPostExecute(result);
         //update user interface here
    }

}

private Document readDocument(String content) {
    Long timeStart = new Date().getTime();
    TagNode tagNode = new HtmlCleaner().clean(content);

    Document doc = null;
    try {
        doc = new DomSerializer(new CleanerProperties()).createDOM(tagNode);
        return doc;
    } catch (ParserConfigurationException e) {
        e.printStackTrace();
    } catch (Exception e) {
        e.printStackTrace();
    }

    return doc;
}

to run the code above use

new getStationsClass.execute();

Answered By - elemianus