Shapefile Support

The shapefile module provides comprehensive support for the ESRI Shapefile format, one of the most widely used geospatial vector data formats. This document explains how the shapefile module works in GeoTools, including its components, usage patterns, and advanced features.

Shapefile Format Overview

A shapefile is actually a collection of files that together store geometric shapes and attribute data. Each shapefile consists of at least three mandatory files with the same basename but different extensions:

Extension	Description	Required
.shp	Contains the geometry data	Yes
.shx	Index file with positional information for the geometry	Yes
.dbf	dBASE file containing attribute data	Yes
.prj	Projection definition file in WKT format	No (recommended)
.qix	Optional spatial index for faster queries	No
.fix	Optional feature ID index	No
.cpg	Optional character encoding information	No

100%

Core Architecture

The shapefile module is built around several key classes that work together to provide access to shapefile data:

100%

Main Components

1. ShapefileDataStore: The central class that provides access to the shapefile data. It's an implementation of the GeoTools DataStore interface specifically for shapefiles.

2. ShapefileDataStoreFactory: Creates instances of ShapefileDataStore from parameters.

3. ShpFiles: Manages access to the various files that make up a shapefile.

4. ShapefileSetManager: Handles opening the various file readers/writers.

5. IndexManager: Manages the spatial and feature ID indexes.

6. ShapefileFeatureSource: Provides read access to features in the shapefile.

7. ShapefileFeatureStore: Provides read-write access to features in the shapefile.

8. ShapefileFeatureReader: Reads features from a shapefile.

9. ShapefileFeatureWriter: Writes features to a shapefile.

Reading Shapefiles

Reading from shapefiles involves several components working together to extract geometry and attribute data:

100%

The read process coordinates several file types:

• The .shp file provides the geometry data
• The .shx file indexes the geometry records
• The .dbf file provides attribute data
• The .prj file (if present) provides the coordinate reference system

Sample Code

// Create a ShapefileDataStore
File file = new File("states.shp");
Map<String, Object> params = new HashMap<>();
params.put(ShapefileDataStoreFactory.URLP.key, file.toURI().toURL());
ShapefileDataStore dataStore = (ShapefileDataStore) new ShapefileDataStoreFactory().createDataStore(params);

// Get the feature source and features
String typeName = dataStore.getTypeNames()[0];
SimpleFeatureSource source = dataStore.getFeatureSource(typeName);
SimpleFeatureCollection features = source.getFeatures();

// Iterate through the features
try (SimpleFeatureIterator iterator = features.features()) {
    while (iterator.hasNext()) {
        SimpleFeature feature = iterator.next();
        // Process feature...
    }
}

Writing Shapefiles

Creating and writing to shapefiles involves a similar set of components:

100%

The write process creates or updates several files:

• The .shp file stores the geometry data
• The .shx file stores the index to the geometry records
• The .dbf file stores the attribute data
• The .prj file stores the coordinate reference system

Sample Code

// Creating a new shapefile
File file = new File("output.shp");
ShapefileDataStore store = new ShapefileDataStore(file.toURI().toURL());

// Define the feature type
SimpleFeatureTypeBuilder builder = new SimpleFeatureTypeBuilder();
builder.setName("Location");
builder.setCRS(DefaultGeographicCRS.WGS84);
builder.add("the_geom", Point.class);
builder.add("name", String.class);
builder.add("count", Integer.class);
SimpleFeatureType featureType = builder.buildFeatureType();

// Create the schema (creates the empty shapefile)
store.createSchema(featureType);

// Get a feature writer
FeatureWriter<SimpleFeatureType, SimpleFeature> writer = 
    store.getFeatureWriterAppend(store.getTypeNames()[0], Transaction.AUTO_COMMIT);

// Write features
SimpleFeature feature = writer.next();
feature.setAttribute("the_geom", point);
feature.setAttribute("name", "Test Point");
feature.setAttribute("count", 1);
writer.write();
writer.close();

Shapefile and DBF File Limitations

Shapefiles have inherent limitations that the module addresses:

1. File Size Limits:

   • SHP files: Limited to 2GB (due to 32-bit offsets)
   • DBF files: Limited to 4GB (though many systems can only handle 2GB)

2. Field Name Restrictions:

   • Limited to 10 characters in DBF
   • Duplicate names are handled by appending numbers

3. Attribute Type Restrictions:

   • Only supports basic types: String, Number, Date, Boolean
   • String fields limited to 254 characters

4. Geometry Type Restrictions:

   • A shapefile can only contain one type of geometry
   • Multi-geometry support varies by type

ShapefileDumper

The ShapefileDumper utility helps overcome shapefile limitations by splitting data into multiple files:

100%

Key features:

• Handles different geometry types by creating separate shapefiles
• Splits large datasets into multiple files with sequential naming
• Controls shapefile size limits through configuration
• Sets character encoding for DBF files

Configuration Options

The module offers several configuration options:

Encoding Support

Character encoding for DBF files:

// Setting the charset when creating the datastore
params.put(ShapefileDataStoreFactory.DBFCHARSET.key, Charset.forName("UTF-8"));

// Or after creation
shapefileDataStore.setCharset(Charset.forName("UTF-8"));

// Enable automatic detection from CPG file
shapefileDataStore.setTryCPGFile(true);

Indexing

Control spatial and feature ID indexing:

// When creating the datastore
params.put(ShapefileDataStoreFactory.CREATE_SPATIAL_INDEX.key, Boolean.TRUE);
params.put(ShapefileDataStoreFactory.ENABLE_SPATIAL_INDEX.key, Boolean.TRUE);

// After creation
shapefileDataStore.setIndexed(true);          // Use existing index
shapefileDataStore.setIndexCreationEnabled(true);  // Create if missing
shapefileDataStore.setFidIndexed(true);       // Use FID indexing

Memory Mapping

Enhance performance with memory mapping:

// When creating the datastore
params.put(ShapefileDataStoreFactory.MEMORY_MAPPED.key, Boolean.TRUE);
params.put(ShapefileDataStoreFactory.CACHE_MEMORY_MAPS.key, Boolean.TRUE);

// After creation
shapefileDataStore.setMemoryMapped(true);
shapefileDataStore.setBufferCachingEnabled(true);

Reading and Writing Process Flow

The shapefile module coordinates reading and writing across multiple files:

100%

Technical Details

Geometry Type Mapping

Shapefile and JTS geometry type mapping:

Shapefile Type	JTS Geometry Types
POINT	Point
MULTIPOINT	MultiPoint
ARC (POLYLINE)	LineString, MultiLineString
POLYGON	Polygon, MultiPolygon

DBF Attribute Types

Supported DBF attribute types and their mappings:

DBF Type	Description	Java Type
C	Character	String
N	Numeric	Integer, Long, or Double
F	Float	Double
L	Logical	Boolean
D	Date	Date
@	Timestamp	Timestamp

Integration with GeoTools

The shapefile module integrates with the wider GeoTools ecosystem:

1. Implements the DataStore API for consistent access to different data sources
2. Uses JTS (Java Topology Suite) for geometry representation
3. Supports GeoTools' CRS (Coordinate Reference System) framework
4. Enables filtering and querying using GeoTools Filter API

Performance Considerations

For optimal performance with shapefiles:

1. Use spatial indexing: Enable QIX indexing for faster spatial queries
2. Use FID indexing: Enable FIX indexing for faster feature ID lookups
3. Memory mapping: Enable for larger files when sufficient memory is available
4. Buffer caching: Cache buffers to reduce I/O operations
5. Charset handling: Only specify non-default character sets when needed