Getting Started with streamConnect
Michael Hahsler
Source:vignettes/streamConnect.Rmd
streamConnect.RmdstreamConnect connects data stream sources and stream
mining algorithms through sockets or web services. It works with the
DSD (data stream) and DSC (data stream
clustering) interfaces from the stream package.
This guide shows two common workflows: reading a stream over a socket
and updating a remote clusterer through a web service. For more control
over the web service endpoints and deployment, see the help pages for
publish_DSD_via_WebService() and
publish_DSC_via_WebService().
Install and load
Install the released package from CRAN:
install.packages("streamConnect")Load the package and select an available local port for the examples:
#> Loading required package: stream
We will use the same free server port throughout this vignette. Some systems do not allow the use of local ports, then the code in this vignette will not be executed.
run_code <- TRUE
tryCatch({
port <- httpuv::randomPort()
}, warning = function(w) {
message("Ports unavailable: ", w)
run_code <<- FALSE
}, error = function(e) {
message("Ports unavailable: ", e)
run_code <<- FALSE
})Read a data stream over a socket
Start a background process that publishes a stream. The publisher opens a socket server and writes data in blocks. Since the socket format does not include column names, provide them when creating the reader.
socket_process <- publish_DSD_via_Socket(
DSD_Gaussians(k = 3, d = 2),
port = port
)
socket_process#> PROCESS 'R', running, pid 7803.
Connect a local DSD reader and request points from the
stream:
socket_dsd <- DSD_ReadSocket(
port = port,
col.names = c("x", "y", ".class")
)
points <- get_points(socket_dsd, n = 10)
head(points)#> x y .class
#> 1 0.2021916 1.00086784 2
#> 2 0.2147846 0.95420599 2
#> 3 0.4085599 0.60682598 1
#> 4 0.2032706 0.92301402 2
#> 5 0.2011201 0.93230287 2
#> 6 0.3822066 0.06442653 3
Close the reader and stop the publisher when finished:
close_stream(socket_dsd)
if (socket_process$is_alive()) socket_process$kill()#> [1] TRUE
Update a remote clusterer via a web service
Web services also support request and response operations. Start a
clusterer in a background process, then create a local DSC
interface for it:
clusterer_process <- publish_DSC_via_WebService(
"DSC_DBSTREAM(r = .05)",
port = port
)
clusterer <- DSC_WebService(paste0("http://127.0.0.1:", port))
clusterer#> Web Service Data Stream Clusterer: DBSTREAM
#> Served from: http://127.0.0.1:11336
#> Class: DSC_WebService, DSC_R, DSC
#> Number of micro-clusters: 0
#> Number of macro-clusters: 0
Send a batch of points to the remote clusterer and retrieve its centers and weights through the local interface:
stream <- DSD_Gaussians(k = 3, d = 2, noise = 0.05)
update(clusterer, stream, n = 500)
get_centers(clusterer)#> # A tibble: 24 × 2
#> X1 X2
#> <dbl> <dbl>
#> 1 0.332 0.292
#> 2 0.470 0.933
#> 3 0.634 0.120
#> 4 0.271 0.288
#> 5 0.232 0.314
#> 6 0.453 0.886
#> 7 0.601 0.0798
#> 8 0.700 -0.00260
#> 9 0.644 0.0666
#> 10 0.652 0.00159
#> # ℹ 14 more rows
get_weights(clusterer)#> [1] 50.364869 62.925534 17.643684 94.970135 62.171627 57.319254 13.376050
#> [8] 28.404639 60.600800 29.533323 30.663367 48.203102 13.982802 5.231117
#> [15] 12.220861 8.834957 33.911419 45.884032 35.707734 27.142056 6.202858
#> [22] 8.310432 30.521944 6.727164
Stop the service process when the work is complete:
if (clusterer_process$is_alive()) clusterer_process$kill()#> [1] TRUE
Publish a stream through a web service
To expose a data source over HTTP, publish a DSD and
read it with DSD_ReadWebService(). The service returns CSV
by default; JSON and RDS serialization are also available through the
serializer argument.
source_process <- publish_DSD_via_WebService(
"DSD_Gaussians(k = 3, d = 2)",
port = port
)
remote_stream <- DSD_ReadWebService(paste0("http://127.0.0.1:", port))
get_points(remote_stream, n = 10)#> # A tibble: 10 × 3
#> X1 X2 .class
#> <dbl> <dbl> <dbl>
#> 1 0.437 0.908 1
#> 2 0.440 0.0144 3
#> 3 0.199 0.524 2
#> 4 0.120 0.593 2
#> 5 0.0950 0.538 2
#> 6 0.0656 0.593 2
#> 7 0.483 -0.000592 3
#> 8 0.225 0.507 2
#> 9 0.354 0.885 1
#> 10 0.414 0.897 1
if (source_process$is_alive()) source_process$kill()#> [1] TRUE
Next steps
- Use
publish_DSD_via_Socket()andDSD_ReadSocket()for continuous point-to-point stream transfer. - Use
publish_DSD_via_WebService()andDSD_ReadWebService()to serve stream data over HTTP. - Use
publish_DSC_via_WebService()andDSC_WebService()to update and inspect a remote clusterer. - Set
serve = FALSEto write a plumber task file for deployment with a hosting environment of your choice.
See the package reference for the full function documentation.