Pagination is not bulletproof. If the results you are paging from are changed, you can miss items or receive duplicate items. If you absolutely must have every item, don’t use paging.
The problem is that data changes while you are paging through it. For example, the requester asks for the first 10 records. You return records 1 through 10. Then someone deletes record 2. The requester then asks for the second 10 records. The first 10 records are now records 1 and 3 through 11 (since record 2 has been deleted). The second 10 would then be records 12 through 22. Effectively, the person paging through the data has missed record 11 without knowing it.
Certain calls made to Candlepin can be given parameters that will cause Candlepin to return the results back in pages. Currently those calls are
You can specify four parameters that affect paging.
The order and sort_by options can alternately be specified without the page and per_page parameters. In this case, all the results will be returned sorted in the manner specified by the parameter values.
When the page and per_page parameters are specified, a special header named
Link is returned with the response. The URLs within the Link header provide
navigation to the first, next, previous, and last pages. Please see RFC
5988 for a full specification of the Link
header. The header provides a list of links separated by commas. Each link is
split into a URL (surrounded by angle brackets) and its relation defined by
the string rel="RELATION_NAME"
. The four relation names are first
, next
,
prev
, and last
. A simple parser for the Link header can be found
here.
You should use the Link header to navigate rather than trying to craft URLs
yourself.
RestEASY sends every request through the PageRequestFilter which examines the request’s query string for the parameters specified above. It takes the values of these parameters and sets them in an object called a PageRequest. After dealing with the various cases of when to use defaults, the PageRequest is then placed in the context.
In order to add paging to a resource, you must modify the resource method to
take and read the PageRequest object. This is as simple as adding @Context
PageRequest pageRequest
to the parameter list in the method signature. Now you
need to do your paging magic (more on this later) and create a Page object. The
page object has three fields that must be set: the actual page data (a Java
Collection), the maximum number of records, and the PageRequest that was sent
into the method. You then place the Page object into the context with a
ResteasyProviderFactory.pushContext(Page.class, page);
and return the page data of the page. The Page object must be placed in the context so that the LinkHeaderResponseFilter can have access to the paging information to build the navigation links.
Here’s a simple example of Resource method that has pagination enabled.
@GET
@Produces(MediaType.APPLICATION_JSON)
@Wrapped(element = "consumers")
public List<Consumer> list(@QueryParam("username") String userName,
@QueryParam("type") String typeLabel,
@QueryParam("owner") String ownerKey,
@Context PageRequest pageRequest) {
ConsumerType type = null;
if (typeLabel != null) {
type = lookupConsumerType(typeLabel);
}
Owner owner = null;
if (ownerKey != null) {
owner = ownerCurator.lookupByKey(ownerKey);
if (owner == null) {
throw new NotFoundException(
i18n.tr("Organization with key: {0} was not found.",
ownerKey));
}
}
// We don't look up the user and warn if it doesn't exist here to not
// give away usernames
Page<List<Consumer>> p = consumerCurator.listByUsernameAndType(userName,
type, owner, pageRequest);
// Store the page for the LinkHeaderPostInterceptor
ResteasyProviderFactory.pushContext(Page.class, p);
return p.getPageData();
}
Earlier I mentioned that some paging magic must occur. This magic occurs in
the Hibernate layer. There are two methods, listAll()
and listByCriteria()
in the AbstractHibernateCurator that have signatures that accept PageRequest
objects. These methods are written to examine the PageRequest object and create
a resultant Page object. Ideally, the best way is to have a method in your
curator that builds a DetachedCriteria object with your necessary filters. Then
send that DetachedCriteria and the PageRequest into listByCriteria()
. You
will receive a Page object back.
What if you need to perform some filtering of the data after you read it back from the database? So far, my solution has been to read all the data, filter it, take a sublist of the correct size, and throw the rest of the data away. This may sound wasteful, but it is still faster than returning all the results in JSON as the serialization of the objects to JSON takes a significant amount of time.
There are a few things to keep in mind if you must perform post-read filtering and wish to use the magic methods in AbstractHibernateCurator
listAll()
or listByCriteria()
method that takes a
boolean parameter called postFilter
. Since you will be doing
post-filtering, pass in true. With a true value, the method will send in the
PageRequest’s sortBy and order values to the normal listAll()
or
listByCriteria()
. They do this by creating a dummy PageRequest with these
values set and the paging values left null
. It also sets the PageRequest
back to the true pageRequest after making the call to listAll()
or
listByCriteria()
since otherwise the magic method will set it to the dummy.You must be careful when taking the sublist. Use the following method in AbstractHibernateCurator.
public List<E> takeSubList(PageRequest pageRequest, List<E> results) {
int fromIndex = (pageRequest.getPage() - 1) * pageRequest.getPerPage();
if (fromIndex >= results.size()) {
return new ArrayList<E>();
}
int toIndex = fromIndex + pageRequest.getPerPage();
if (toIndex > results.size()) {
toIndex = results.size();
}
// sublist returns a portion of the list between the specified fromIndex,
// inclusive, and toIndex, exclusive.
return results.subList(fromIndex, toIndex);
}
maxRecords
to the number of records in the sublist and don’t forget to set the pageData
to the sublist you create!