Packages

c

org.apache.spark.sql.util

MapperRowCounter

class MapperRowCounter extends AccumulatorV2[Long, List[(Integer, Long)]]

An AccumulatorV2 counter for collecting a list of (mapper index, row count).

Source
MapperRowCounter.scala
Since

3.4.0

Linear Supertypes
AccumulatorV2[Long, List[(Integer, Long)]], Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. MapperRowCounter
  2. AccumulatorV2
  3. Serializable
  4. Serializable
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new MapperRowCounter()

Value Members

  1. def add(v: Long): Unit

    Takes the inputs and accumulates.

    Takes the inputs and accumulates.

    Definition Classes
    MapperRowCounterAccumulatorV2
  2. def copy(): MapperRowCounter

    Creates a new copy of this accumulator.

    Creates a new copy of this accumulator.

    Definition Classes
    MapperRowCounterAccumulatorV2
  3. def copyAndReset(): MapperRowCounter

    Creates a new copy of this accumulator, which is zero value.

    Creates a new copy of this accumulator, which is zero value. i.e. call isZero on the copy must return true.

    Definition Classes
    MapperRowCounterAccumulatorV2
  4. final def id: Long

    Returns the id of this accumulator, can only be called after registration.

    Returns the id of this accumulator, can only be called after registration.

    Definition Classes
    AccumulatorV2
  5. final def isRegistered: Boolean

    Returns true if this accumulator has been registered.

    Returns true if this accumulator has been registered.

    Definition Classes
    AccumulatorV2
    Note

    All accumulators must be registered before use, or it will throw exception.

  6. def isZero: Boolean

    Returns false if this accumulator has had any values added to it or the sum is non-zero.

    Returns false if this accumulator has had any values added to it or the sum is non-zero.

    Definition Classes
    MapperRowCounterAccumulatorV2
  7. def merge(other: AccumulatorV2[Long, List[(Integer, Long)]]): Unit

    Merges another same-type accumulator into this one and update its state, i.e.

    Merges another same-type accumulator into this one and update its state, i.e. this should be merge-in-place.

    Definition Classes
    MapperRowCounterAccumulatorV2
  8. final def name: Option[String]

    Returns the name of this accumulator, can only be called after registration.

    Returns the name of this accumulator, can only be called after registration.

    Definition Classes
    AccumulatorV2
  9. def reset(): Unit

    Resets this accumulator, which is zero value.

    Resets this accumulator, which is zero value. i.e. call isZero must return true.

    Definition Classes
    MapperRowCounterAccumulatorV2
  10. def setPartitionId(id: Integer): Unit
  11. def toString(): String
    Definition Classes
    AccumulatorV2 → AnyRef → Any
  12. def value: List[(Integer, Long)]

    Defines the current value of this accumulator

    Defines the current value of this accumulator

    Definition Classes
    MapperRowCounterAccumulatorV2